Skip to content

Transition tolerance specification to log domain #3556

@gwhitney

Description

@gwhitney

Describe the bug
The range of positive JavaScript number values limits the tolerance parameters for equality testing to roughly 1e-320, whereas when computing with high-precision BigNumbers, a smaller tolerance would be useful. For example, the available digits for some transcendental functions are limited by the tolerance rather than precision, since the iterative algorithms can no longer function once the incremental changes are within the equality tolerance.

To Reproduce
math.config({relTol: 1e-500}) -- oops, results in relative tolerance 0, which is a valid setting but a very different behavior than an actual relative tolerance of 1e-500.

Discussion
The recommended resolution, which has been discussed before although I can't find where, is to transition to two new configuration parameters, which would simply give the digits of tolerance (i.e. be -log_10 of the current values). To avoid breaking change, setting relTol and absTol could still work, with a deprecation warning, much like epsilon, which should then be retired altogether in the next breaking change.

All that's really needed to move forward is a choice of the new parameter names:

  • absoluteDigits and relativeDigits?
  • absTolDigits and relTolDigits?
  • logAbsTol and logRelTol?

None of these are super euphonious. @josdejong should just pick something, one of these or something better he thinks of.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions