db/state: restore empty tombstones on unwind and honor them in getLatestFromDb#20627
Conversation
files do encode empty byte as marker for deletion. this is different from absence |
|
Fix #1 relies on the
Both empty cases have
diffs := []kv.DomainEntryDiff{{Key: string(k) + string(stepBytes), Value: nil}}
require.NoError(dt.unwind(ctx, tx, step, txNumUnwindTo, currentFilesEndStep, diffs))
// Expect: no (k, unwindStep, ...) entry — nil means "different step, skip"That pins the asymmetry into the test suite so a future refactor can't silently flip the semantics back. |
Fix the DB-level unwind path so first-time-write reversals leave a proper
empty-value tombstone in the domain values table, rather than deleting the
current-step entry and silently falling back to files (which have no
concept of deletions and therefore return stale pre-unwind data).
`DomainEntryDiff.Value` carries three distinct shapes that must be handled
differently during unwind (per the comment in db/kv/helpers.go):
- nil — "different step": prev value lives at another step, skip
restore (only produced by legacy V0 changesets where
valueLen==0 deserializes as nil)
- []byte{} — "no previous value": key was absent before this step; write
an empty tombstone at the unwind step so subsequent reads
see the key as absent again
- non-empty — restore the actual previous value
The old guard `if len(value) > 0` conflated `nil` and `[]byte{}` and
skipped both, leaving no tombstone after unwinding a first-time write.
Corrected to `if value != nil`, matching the diff's documented contract.
The written tombstone lands at max(step, currentFilesEndStep), which by
construction is beyond the frozen file range, so the existing step-age
guard in getLatestFromDb already accepts it without needing to change the
read path.
This is a deliberately narrow re-land of #20483 (reverted by #20509). The
PR's other two pieces — the `len(v)==0` empty-authority bypass in
getLatestFromDb and the LargeValues cross-check in getLatest — are NOT
included here. Regressions surfaced in kurtosis/assertoor_glamsterdam
(which runs with --experimental.bal and the storagerefundtx spammer,
generating a deluge of tombstones) suggest the empty-authority bypass
has downstream interactions with the BAL / post-Amsterdam paths that
need separate narrowing. The interrupted-prune LargeValues case is the
original regression suspect from #20509 and remains out-of-scope.
Tests in db/state/domain_test.go:
- TestDomain_UnwindRestoresDeletionMarker (DupSort + LargeValues) — write
a key, delete it, re-write within the same step, build files, then unwind
the re-write. Fails on pre-fix code (getLatest returns the stale post-
unwind value from the frozen file); passes with this fix. Directly
exercises the write-side bug.
- TestDomain_DeletedKeyNotResurrectedByFiles (DupSort + LargeValues) —
retained as a forward regression guard for the read-side contract,
even though it passes on both pre-fix and post-fix code on current main
(file-build + prune semantics have evolved since #20483 such that the
original scenario no longer triggers the read-side bug on its own).
Tested: go test -short ./db/state/... all pass; make lint 0 issues;
make erigon builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
760387c to
c447697
Compare
|
Force-pushed a narrowed version of this PR to address the recurring kurtosis/assertoor_glamsterdam CI failure. Summary of the narrowing:
Why: The glamsterdam suite is the one kurtosis configuration that runs with The unwind fix alone is sufficient for the originally-reported bug (stale reads after unwind on post-Fusaka mainnet), because the tombstone is written at
Out of scope, tracked for follow-up:
CI will re-run on the force-push — expecting glamsterdam to return to ~12min passing. |
|
Confirmed — the narrowing works. Local reproduction (kurtosis 1.18.1, ethereum-package 6.1.0, same glamsterdam.io config as CI): Total wall time from enclave CI also green on the amended commit: So the Follow-up work captured in the commit body:
|
|
CI Gate is not happy: https://github.com/erigontech/erigon/actions/runs/24597293981/job/71929690553 |
Summary
Re-land the narrow core of #20483 (reverted by #20509), addressing the DB-layer siblings of the post-unwind stale-read bug. Complements #20625 which addressed the in-memory overlay side.
Two linked bugs in the DB-level domain unwind and read paths caused stale data resurrection after an unwind that reverted a first-time write or a deletion inside the unwound range.
Symptom observed on mainnet
Post-Fusaka mainnet catch-up sync with a chaindata/ wipe (snapshots/ retained). On the first re-executed block after a forkchoice-driven unwind, execution returned less gas than the header — diffs observed of
-34200(block 24,898,955),-73829(block 24,899,403),-118872(block 24,899,594). The diffs break down into multiples ofSSTORE_SET - SSTORE_RESET = 17100plus cold-access flips of2600.Previous PR #20625 cleared the first two by pruning the in-memory overlay on Unwind. Block 24,899,594 still failed because the overlay was already flushed to DB at Unwind time — the stale-read path now was purely DB-layer, addressed here.
Fixes
1.
unwind()must restore empty tombstones —db/state/domain.go:1317(both DupSort and LargeValues paths)DomainEntryDiff.Valuehas three shapes, documented indb/kv/helpers.go:247:nil— "different step": prev value lives at another step, skip restore (legacy V0 changeset shape)[]byte{}— "no previous value": key was absent before this step; write an empty tombstone so the key appears absent again after the unwind completesThe old guard
if len(value) > 0skipped bothniland[]byte{}, leaving no tombstone after unwinding a first-time write. Corrected toif value != nil.2.
getLatestFromDbmust treat empty values as authoritative —db/state/domain.go:1665Empty-value entries are deletion tombstones. The step-age guard previously discarded them when their step fell within the frozen file range, causing the caller to fall through to
getLatestFromFiles. Frozen files encode deletions as absence, so the file returns the pre-deletion value — the exact resurrection the deletion was meant to prevent. Empty entries are now returned asfound=trueregardless of step age; the step-age guard still applies to non-empty entries.Relationship to #20483 / #20509
This is a deliberate re-land of the narrow core of #20483. Key differences:
getLatest(PR db/state: fix stale slot resurrection after unwind and deletion #20483 lines 1697–1737). That handled an interrupted-PruneSmallBatchesedge case specific toCodeDomain/RCacheDomainand was the most likely source of the regressions that motivated revert "db/state: fix stale slot resurrection after unwind and deletion (#20483)" #20509's revert. If it proves necessary, it can be added later as a separate PR with its own dedicated test.Tests
TestDomain_UnwindRestoresDeletionMarker(DupSort + LargeValues subtests) — writes a key, deletes it, re-writes within the same step, builds files, then unwinds the re-write. Fails on pre-fix code (getLatest returns the stale post-unwindvalue2from the frozen file); passes with the fix. Exercises the write-side bug directly.TestDomain_DeletedKeyNotResurrectedByFiles(DupSort + LargeValues subtests) — documents the read-side contract by writing a key and deleting it at a step that falls within file range. Passes on currentmaineven without the fix (the file-build + prune semantics evolved since db/state: fix stale slot resurrection after unwind and deletion #20483 and no longer hit the specific stale-read in this exact test scenario), but retained as a forward regression guard and as documentation of the invariant.Test plan
go test -short ./db/state/...— all passmake lint— 0 issuesmake erigon— builds clean🤖 Generated with Claude Code