Skip to content

Negative variance in result of division by int32 #3236

@SimonHeybrock

Description

@SimonHeybrock
import scipp as sc

a = sc.scalar(419.0, variance=419.0)
b = sc.scalar(1615722, dtype='int32')
a/b  # variance = -5.35439526e-07

A brief look shows that we are computing b * b as an intermediate steps, which overflows in int32.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions