-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently decimal arithmetic may return an overflow error if an arithmetic operation produces an output that exceeds the maximum precision of the storage type.
Describe the solution you'd like
Spark and similar systems instead truncate the precision of the output, if the precision would exceed the maximum of the storage type - https://github.com/apache/arrow/blob/36ddbb531cac9b9e512dfa3776d1d64db588209f/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/DecimalTypeUtil.java#L83
To achieve this the arithmetic kernels would need to be updated to detect this case, and instead perform arithmetic on double-width decimal primitives, before then truncating this down to an appropriate scale/precision.
Describe alternatives you've considered
We could continue to return an error, but this is likely to be surprising for users.
Additional context
#4640 proposes increasing the decimal output scale, which in turn would make this error case more likely.
The fundamental primitive necessary for this is, N-digit division, which will likely be implemented as part of #4663