Skip to content

fix: fix precision loss for large decimal values#4135

Merged
sidorares merged 3 commits intomasterfrom
fix/decimal-precision-loss-for-large-values
Mar 2, 2026
Merged

fix: fix precision loss for large decimal values#4135
sidorares merged 3 commits intomasterfrom
fix/decimal-precision-loss-for-large-values

Conversation

@sidorares
Copy link
Owner

Fix precision loss for large decimal values

Fixes #3690
Fixes #2928

Background

MySQL2 uses manual parsing in Packet.parseFloat() to avoid expensive Buffer.toString() conversions. This optimization provides ~3x performance improvement (see comment in packet.js L27-36) by parsing numbers directly from buffer bytes.

However, the current implementation accumulates floating-point rounding errors with repeated *10 operations, causing precision loss for numbers with many digits.

Issues Fixed

Issue #3690: DECIMAL(36,18) Precision Loss

Problem: Values like 50000.000000000000000000 (DECIMAL(36,18) with 18 fractional digits) were parsed incorrectly:

// Before:
"50000.000000000000000000"  49999.99999999999  

// After:
"50000.000000000000000000"  50000  

Issue #2928: MAX_VALUE Doubles Corruption

Problem: Very large DOUBLE values lost precision in the last significant digits:

// Before:
"1.7976931348623157e308"  1.7976931348623155e+308  
"-1.7976931348623157e308"  -1.7976931348623155e+308  

// After:
"1.7976931348623157e308"  1.7976931348623157e+308  
"-1.7976931348623157e308"  -1.7976931348623157e+308  

Solution

Add a length-based bailout to parseFloat() for numbers >17 characters:

if (len > 17) {
  const str = this.buffer.toString('utf8', this.offset, this.offset + len);
  this.offset += len;
  return Number.parseFloat(str);
}

Why 17? IEEE 754 double precision supports ~15-17 significant digits. Testing shows this threshold:

  • Avoids precision loss from accumulated errors
  • Affects only ~1% of typical MySQL data
  • Preserves fast path for 98%+ of common cases

Performance Impact

Tested with 50,000 realistic MySQL values:

  • 98.9% of data uses fast path (≤17 chars) - unchanged performance
  • 1.1% uses parseFloat bailout (>17 chars) - fixes precision issues
  • Overall throughput: 13.7M ops/sec (minimal impact)

The fast path remains for:

  • All typical integers (IDs, counts, years)
  • Most decimals (prices, measurements)
  • Common numeric fields

Testing

Added comprehensive test suite with 54 test cases:

See: test/unit/packets/test-packet-parseFloat-precision.test.mts

Alternative Approaches Explored

WebAssembly Implementation

Explored two WASM approaches as potential solutions:

  1. Happy Path: WASM for common cases, bailout for complex
  2. Complete Safe: Handle everything in WASM with safe precision

Benchmark Results (50,000 test cases):

  • JavaScript (with fix): 31.1M ops/sec ⭐
  • WASM Happy Path: 17.1M ops/sec
  • WASM Complete: 13.8M ops/sec
  • Baseline parseFloat: 12.8M ops/sec

Outcome: JavaScript is 40-50% faster than WASM due to:

  • Memory copy overhead (50-100ns per call)
  • WASM call boundary overhead (30-60ns)
  • Excellent V8 JIT optimization for this workload

Conclusion: WASM not beneficial for this use case. The minimal JavaScript fix is optimal.

References

@codecov
Copy link

codecov bot commented Feb 27, 2026

Codecov Report

❌ Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.28%. Comparing base (ff8ed8a) to head (e5dd7af).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
lib/packets/packet.js 91.30% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4135      +/-   ##
==========================================
+ Coverage   90.23%   90.28%   +0.05%     
==========================================
  Files          86       86              
  Lines       13740    13757      +17     
  Branches     1664     1668       +4     
==========================================
+ Hits        12398    12421      +23     
+ Misses       1342     1336       -6     
Flag Coverage Δ
compression-0 89.51% <91.30%> (+0.05%) ⬆️
compression-1 90.26% <91.30%> (+0.05%) ⬆️
static-parser-0 87.88% <91.30%> (+0.05%) ⬆️
static-parser-1 88.64% <91.30%> (+0.05%) ⬆️
tls-0 89.69% <91.30%> (+0.05%) ⬆️
tls-1 90.06% <91.30%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sidorares sidorares force-pushed the fix/decimal-precision-loss-for-large-values branch 2 times, most recently from d971e2f to 0cee994 Compare February 27, 2026 06:15
@sidorares sidorares marked this pull request as draft February 27, 2026 07:02
@wellwelwel wellwelwel changed the title Fix precision loss for large decimal values fix: fix precision loss for large decimal values Feb 27, 2026
Add length-based bailout to parseFloat() to fix accumulated rounding
errors from repeated *10 operations. For numbers longer than 17
characters, delegate to Number.parseFloat() which handles precision
correctly.

This fixes two critical issues:
- DECIMAL(36,18) precision loss where 50000.000...0 parsed as 49999.999
- MAX_VALUE doubles corruption where last digits were incorrect

The threshold of 17 is based on IEEE 754 double precision limits
(~15-17 significant digits). Testing shows this affects only ~1% of
typical MySQL data while preserving the fast path for 98%+ of cases.

Add comprehensive test suite with 54 test cases covering both issues,
edge cases, and regression tests.

Closes #3690
Closes #2928
@sidorares sidorares force-pushed the fix/decimal-precision-loss-for-large-values branch from 0cee994 to 5f16d62 Compare February 27, 2026 09:58
Add tests exercising parseFloat bailout paths to improve coverage:
- DECIMAL(36,18) with many fractional digits (>17 chars)
- DOUBLE with scientific notation values

These integration tests ensure the bailout conditions are covered
in real database query scenarios.
@sidorares sidorares force-pushed the fix/decimal-precision-loss-for-large-values branch from 1934664 to 4202e21 Compare February 27, 2026 10:39
@sidorares sidorares marked this pull request as ready for review February 28, 2026 09:44
@sidorares sidorares merged commit 099beea into master Mar 2, 2026
88 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mariadb decimalNumbers issue Compatibility issue - Possible corruption of values with min and max values for a double.

2 participants