Investigation Findings

Blocks with commitment root mismatches found via check-commitment-hist-at-blk. Correct values confirmed by comparing dump-hist-at-blk output against debug_traceBlockByNumber (prestateTracer + diffMode) from Alchemy. Erigon's own execution traces match Alchemy — the bug is in the persisted history, not in execution.

Block 549767 (earliest known)

1 storage mismatch:

Plus cascading balance mismatch on 0x580992b51e3925e23280efb93d3047c82f17e038.

Block 15501811

7 account mismatches + 16 storage mismatches.

Account mismatches (nonce/balance):

Plus balance mismatches on these 4 accounts and 3 others (WETH, 0x283af..., 0xf326e...).

Storage mismatches:

Block 15523659

1 storage mismatch:

Overriding this single value with check-commitment-with-override produces a matching commitment root.

Full EF sequence for this slot (raw HistorySeek values — history stores the OLD value before each write):

txNum            block        step     rawValue (old)       GetAsOf+1 (after)
1739356862       15523655     4452     0x0                  0x01                 <- correct
1739357208       15523656     4452     0x01                 0x02                 <- correct
1739357596       15523658     4452     0x02                 0x03                 <- correct
1739357842       15523659     4452     0x03                 0x02                 <- raw correct, GetAsOf wrong!
1739370674       15523735     4452     0x02                 <not_found>          <- RAW WRONG (should be 0x04)

• Raw value at txNum 1739357842 is correct (0x03 = old value before 3->4 write)
• Corruption is at txNum 1739370674 (block 15523735, a deletion): raw value is 0x02 instead of 0x04
• GetAsOf for block 15523659's end state resolves through the wrong deletion entry

Theories investigated

• Write-time prevVal bug: ruled out — fresh re-execution from block 0 produces correct history, corruption appears after merge
• Missing recalcVisibleFiles after merge: ruled out — cleanAfterMerge does call it (aggregator.go:1726)
• stateCache returning stale values: ruled out — DB is authoritative after commit
• Paging mismatch between collation and merge: under investigation — collation uses ValuesOnCompressedPage(0) while merge uses default config with paging enabled

Debug tools built

All on branch dbg_commitment_error:

• erigon seg dump-hist-at-blk — dump all touched key values at block end
• erigon seg dump-slot-history — deep-dive into a single key's EF entries and raw history values
• erigon seg check-commitment-with-override — verify commitment root with patched values
• erigon seg find-deleted-slots — scan for deleted storage slots (potential bug candidates)
• cmd/scripts/checkpoint_debug/compare_hist_diff.py — compare dump against RPC trace