Skip to content

db/kv, db/migrations, cmd: drop pre-E3 plain/hashed state and history tables#21280

Merged
AskAlexSharov merged 5 commits into
mainfrom
yperbasis/drop_legacy_tables
May 22, 2026
Merged

db/kv, db/migrations, cmd: drop pre-E3 plain/hashed state and history tables#21280
AskAlexSharov merged 5 commits into
mainfrom
yperbasis/drop_legacy_tables

Conversation

@yperbasis

@yperbasis yperbasis commented May 19, 2026

Copy link
Copy Markdown
Member

Summary

Schema-cleanup half of #12440 (incarnation removal). Seven pre-E3 ("E2-era") tables are dormant in production — they are not read by the canonical E3 execution / commitment path, which works off kv.AccountsDomain / kv.StorageDomain / kv.CommitmentDomain:

  • PlainState — account + storage rows; storage keys baked the per-account incarnation counter into [addr]+[inc]+[key]
  • HashedAccount / HashedStorage — legacy MPT inputs; storage keys used [hash(addr)]+[inc]+[hash(key)]
  • AccountChangeSet / StorageChangeSet — per-block reverse-diff storage with [blockNum]+[addr](+inc) keys
  • AccountHistory / StorageHistory — roaring-bitmap E2 history indices, superseded by the AccountHistory* / StorageHistory* domain tables

This PR moves all seven from kv.ChaindataTables into kv.ChaindataDeprecatedTables so the live schema no longer recreates them on fresh DBs. A new drop_legacy_e2_tables migration drops the buckets (both data and metadata) on existing E2-migrated databases; the migration is idempotent — DropTable is a no-op for tables that were never materialised, so fresh-DB upgrades are a pass-through.

Knock-on cleanups required because the now-deprecated tables would otherwise still be read/written:

  • t8ntool.CalculateStateRoot — dropped the dead PlainState → HashedStorage/HashedAccounts hashing loop; ComputeCommitment reads from kv.CommitmentDomain directly.
  • kvcache.AssertCheckValues (assert-tag-only debug check that compared cache rows against kv.PlainState) — removed, along with its sole call site in txnprovider/txpool/pool.go.
  • cmd/integration/commands/refetence_db.go and cmd/pics/state.go — dropped deprecated tables from their debug whitelists.
  • cmd/rpctest/account_range_verify used kv.E2AccountsHistory purely as a bucket name to JSON-dump eth_accountRange results into a temp MDBX env for erigon/geth comparison; switched to a local AccountDump bucket via mdbx.WithTableCfg so the temp DB is fully decoupled from the chain schema.
  • TestManagedTx (db/kv/mdbx) — was picking the first two entries of the alphabetically-sorted ChaindataTables list and relying on them having matching DupSort flags; switched to explicit TblAccountVals / TblStorageVals.
  • TestEthCallToPrunedBlock (rpc/jsonrpc) — dropped the now-impossible PruneTableDupSort(kv.StorageChangeSetDeprecated, ...) call.
  • TestEviction (db/kv/kvcache) — dropped dead tx.Put(kv.PlainState, ...) calls that the test's assertions never depended on.

This spinoff is independent of and lands ahead of the in-memory IncarnationPath removal (already on main) and the storage-key + legacy-MPT-path incarnation removal (#21256), so reviewers focused on schema vs trie/state-write changes can evaluate the two in isolation.

Test plan

  • make erigon integration — clean
  • go build ./... — clean (default and -tags assert)
  • make lint0 issues
  • go test -short -failfast ./rpc/jsonrpc/... ./db/kv/... ./db/migrations/... ./cmd/evm/... ./cmd/rpctest/... ./txnprovider/txpool/... — all green, including the previously-failing TestManagedTx and TestEthCallToPrunedBlock
  • CI: tests-mac-linux / race-tests / sonar / kurtosis

🤖 Generated with Claude Code

… tables

E3 execution stores state in kv.AccountsDomain / kv.StorageDomain /
kv.CommitmentDomain, which are incarnation-free. The following pre-E3
("E2-era") tables are dormant in production — they are not read by
the canonical execution / commitment path:

  - PlainState                  (account + storage rows; storage keys
                                 baked the per-account incarnation
                                 counter into [addr]+[inc]+[key])
  - HashedAccount / HashedStorage (legacy MPT inputs; storage keys
                                   used [hash(addr)]+[inc]+[hash(key)])
  - AccountChangeSet / StorageChangeSet (per-block reverse-diff
                                         storage with [blockNum]+[addr]
                                         (+inc) keys)
  - AccountHistory / StorageHistory (roaring-bitmap E2 history indices,
                                     superseded by the AccountHistory*
                                     / StorageHistory* domain tables)

Schema
- All seven constants move from kv.ChaindataTables to
  kv.ChaindataDeprecatedTables. The mdbx migrator opens the buckets
  read-only if they exist (DBAccede) and drops the empty ones on the
  next exclusive open; the live schema no longer recreates them on
  fresh DBs.
- DupSort flags for PlainState / HashedStorage / AccountChangeSet /
  StorageChangeSet are preserved in ChaindataTablesCfg so existing E2
  databases can still be opened with the correct bucket flags by the
  migration before it clears them.

Migration
- New drop_legacy_e2_tables migration (db/migrations/) clears all
  seven tables on existing E2-migrated databases. The migration is
  idempotent — ClearTable is a no-op on already-empty tables.

Caller cleanups
- t8ntool's CalculateStateRoot dropped the PlainState →
  HashedStorage/HashedAccounts hashing loop; ComputeCommitment reads
  from kv.CommitmentDomain directly and the loop's writes were never
  consumed.
- kvcache.AssertCheckValues (an assert-tag-only debug check that
  compared cache rows against kv.PlainState) is stubbed to a no-op —
  the active state for cache validation now lives in
  kv.AccountsDomain / kv.StorageDomain and the read-key format no
  longer matches what the cache stores.
- cmd/integration/commands/refetence_db.go and cmd/pics/state.go drop
  the deprecated tables from their debug whitelists.
- cmd/rpctest's account_range_verify used kv.E2AccountsHistory purely
  as a bucket name to JSON-dump eth_accountRange results into a fresh
  temp MDBX env for erigon/geth comparison (nothing to do with the
  actual E2 history index semantics). Switched the tool to a local
  "AccountDump" bucket via mdbx.WithTableCfg so the temp DB is fully
  decoupled from the chain schema.

Test adjustments
- TestManagedTx (db/kv/mdbx) previously picked kv.ChaindataTables[0]
  and [1] alphabetically and relied on them having matching DupSort
  flags. After the deprecated tables move, that invariant doesn't
  hold (AccountHistory (Flags=0) and AccountHistoryKeys (DupSort)
  now lead the list). Switched the test to explicit TblAccountVals /
  TblStorageVals so it does not depend on sort order.
- TestEthCallToPrunedBlock (rpc/jsonrpc) called PruneTableDupSort on
  kv.StorageChangeSetDeprecated, which the schema no longer
  materialises in ChaindataTables — cursor open panics with
  MDBX_BAD_DBI. Dropped that prune; the active prune on TblAccountVals
  exercises the same code path.

This is the schema-cleanup half of #12440 (incarnation removal) —
the in-memory IncarnationPath removal and the storage-key /
legacy-MPT encoding changes ship separately on top of this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes legacy pre-E3 (“E2-era”) state/history buckets from the active chain DB schema, adds a migration to clear existing instances of those buckets on upgraded databases, and updates tools/tests that previously referenced these deprecated tables.

Changes:

  • Move seven legacy E2 buckets (PlainState / hashed state / changesets / history indices) out of kv.ChaindataTables into kv.ChaindataDeprecatedTables, preventing new DBs from creating them.
  • Add a drop_legacy_e2_tables migration to clear these legacy buckets on existing databases.
  • Remove remaining reads/writes and debug tooling references to the deprecated buckets (t8ntool state root calc, rpctest temp DB dump bucket, integration/pics whitelists, tests).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
rpc/jsonrpc/eth_call_test.go Removes pruning of deprecated changeset bucket; updates test comments.
db/migrations/migrations.go Registers the new drop_legacy_e2_tables migration for ChainDB.
db/migrations/drop_legacy_e2_tables.go New migration to clear legacy E2 buckets.
db/kv/tables.go Moves legacy buckets to ChaindataDeprecatedTables and preserves needed bucket flags in table config.
db/kv/mdbx/kv_abstract_test.go Makes TestManagedTx pick stable DupSort tables instead of relying on ChaindataTables ordering.
db/kv/kvcache/cache.go Stubs AssertCheckValues to a no-op since PlainState is no longer part of the active schema.
cmd/rpctest/rpctest/account_range_verify.go Uses a local-only AccountDump bucket via WithTableCfg instead of reusing E2 history bucket name.
cmd/pics/state.go Removes legacy buckets from debug label map.
cmd/integration/commands/refetence_db.go Removes legacy buckets from debug whitelist.
cmd/evm/internal/t8ntool/transition.go Removes legacy PlainState→hashed-state hashing loop; uses commitment domain directly.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread db/migrations/drop_legacy_e2_tables.go Outdated
Comment thread db/kv/tables.go
Comment thread rpc/jsonrpc/eth_call_test.go
Comment thread rpc/jsonrpc/eth_call_test.go Outdated
…clear

Address Copilot review on #21280:

- drop_legacy_e2_tables migration was calling tx.ClearTable, which only
  empties the bucket (MDBX Drop with del=false) and leaves the bucket
  metadata in place. The doc comments on both the migration and the
  matching ChaindataDeprecatedTables / ChaindataTablesCfg entries said
  "drop" or "clears + drops" — overstated relative to the implementation.
  Switch to tx.DropTable: it requires IsDeprecated=true (which these
  tables now have), reclaims the bucket metadata, and is idempotent —
  dropEvenIfBucketIsNotDeprecated handles the NonExistingDBI case and
  returns nil if a re-open with mdbx.IsNotFound fails.
- eth_call_test.go's doPrune was leaking the time.NewTicker(20s) it
  created for the prune progress log; nothing called Stop() so the
  underlying ticker goroutine survived each test run. Add the missing
  defer.
- Tighten the comment in doPrune to say StorageChangeSetDeprecated is
  no longer "part of the active schema" rather than "materialised in
  the active schema".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
yperbasis added a commit that referenced this pull request May 19, 2026
…clear

Address Copilot review on #21280:

- drop_legacy_e2_tables migration was calling tx.ClearTable, which only
  empties the bucket (MDBX Drop with del=false) and leaves the bucket
  metadata in place. The doc comments on both the migration and the
  matching ChaindataDeprecatedTables / ChaindataTablesCfg entries said
  "drop" or "clears + drops" — overstated relative to the implementation.
  Switch to tx.DropTable: it requires IsDeprecated=true (which these
  tables now have), reclaims the bucket metadata, and is idempotent —
  dropEvenIfBucketIsNotDeprecated handles the NonExistingDBI case and
  returns nil if a re-open with mdbx.IsNotFound fails.
- eth_call_test.go's doPrune was leaking the time.NewTicker(20s) it
  created for the prune progress log; nothing called Stop() so the
  underlying ticker goroutine survived each test run. Add the missing
  defer.
- Tighten the comment in doPrune to say StorageChangeSetDeprecated is
  no longer "part of the active schema" rather than "materialised in
  the active schema".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yperbasis yperbasis requested a review from Copilot May 19, 2026 13:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Comment thread cmd/rpctest/rpctest/account_range_verify.go
Comment thread db/migrations/drop_legacy_e2_tables.go
…ad PlainState test puts

AssertCheckValues was the E2 sanity check that compared cache rows
against kv.PlainState. With PlainState moved to the deprecated-tables
list it could no longer do its job; the previous commit left a no-op
stub so the assert-tag-only call site in txnprovider/txpool/pool.go
would still compile. Just remove both — the stub had no callers outside
this repo and the txpool assertion never observed a real divergence in
practice.

Also drop the `tx.Put(kv.PlainState, ...)` calls in TestEviction: they
silently failed via `_ = tx.Put(...)` and TestEviction's assertions
key off c.OnNewBlock populating the cache directly, not those puts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yperbasis yperbasis requested a review from taratorio as a code owner May 19, 2026 14:04
yperbasis and others added 2 commits May 19, 2026 16:07
The two mdbx envs opened by CompareAccountRange were never closed, so
running the debug tool repeatedly in-process would leak file
descriptors and keep MDBX locks held. Pre-existing in the function but
flagged on the touched lines by the PR review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@AskAlexSharov AskAlexSharov added this pull request to the merge queue May 22, 2026
Merged via the queue into main with commit dfc0333 May 22, 2026
69 checks passed
@AskAlexSharov AskAlexSharov deleted the yperbasis/drop_legacy_tables branch May 22, 2026 10:56
yperbasis added a commit that referenced this pull request May 22, 2026
Five conflicts:

1) cmd/rpctest/rpctest/account_range_verify.go — main added defer
   Close() on the two MDBX handles. Trivial: kept both.

2) db/migrations/migrations.go — main landed #21280 (the legacy E2
   tables drop spinoff originally cut from this branch), which adds
   dropLegacyE2Tables. Moksha has both dropLegacyE2Tables and
   dropAccountIncarnation registered. Resolution: keep both in the
   ChainDB sequence.

3) db/kv/kvcache/cache.go — main removed AssertCheckValues entirely
   (the txpool stub call site went away with #21280's caller cleanup).
   Moksha had it as a no-op stub. Take main's deletion; the only
   reference left was a commented-out test line.

4) execution/stagedsync/exec3_finalize_test.go — PR #21211 deleted
   ~240 lines of test scenarios (TestFinalizeTx_SimpleTransfer,
   _London, _AllScenarios, coinbaseIsRecipientScenario,
   selfTransferScenario, hasCoinbaseDelta, adjustForTransferDelta)
   because the finalizeWithIBS / finalizeTx (delta-args) code paths
   they exercised were dead. Take main's deletion.

5) execution/state/intra_block_state.go — three blocks:

   a) Same-block-revival check in CreateAccount. Main (#21319)
      simplified it to `!account.Empty()` on the version-map-refreshed
      record. Moksha had a path-by-path LatestTxIndex scan
      (BalancePath/NoncePath/CodeHashPath > destructTxIndex). Take
      main's cleaner check — semantically equivalent on the same
      input (the refreshed `account` reflects exactly those higher-
      txIndex writes).

   b) prevInc / IncarnationPath bookkeeping for CreateAccount on
      main. Entirely incarnation-dependent; moksha has no
      IncarnationPath and no Account.Incarnation. Take HEAD's empty
      resolution.

   c) Synthetic versionRead(IncarnationPath, ...) in CreateAccount on
      main. Same reason. Take HEAD's empty resolution.

Plus two non-conflicting touch-ups dropping IncarnationPath references
that the merge silently brought in via main-side changes:

- execution/state/versionmap.go MVReadResultNone/MapRead recursive-
  cross-check whitelist dropped IncarnationPath from the path list.
- execution/state/versionedio.go SD short-circuit zero-value-fallback
  whitelist dropped IncarnationPath from the path list.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
yperbasis added a commit that referenced this pull request Jun 4, 2026
PR #21280 dropped pre-E3 plain/hashed state and history tables (a data migration), so bump DBSchemaVersion and KvServiceAPIVersion to 7.1.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants