db/snapshotsync, kv/prune: skip downloading old commitment-history snapshots by JkLondon · Pull Request #21200 · erigontech/erigon

JkLondon · 2026-05-14T21:25:04Z

Summary

For users running with --prune.include-commitment-history, adds a way to bound how much old commitment-history is downloaded.

Adds --prune.commitment-history.older=<N blocks> — keep commitment history only for the latest N blocks; older commitment-history snapshots are skipped at download time. 0 (default) keeps everything. Requires --prune.include-commitment-history.
Conversion blocks → steps happens once per run inside SyncSnapshots (olderBlocks * maxStateStep / maxBlock), so the user-facing unit is blocks (linked to time) while the internal filter stays in steps.
The filter applies only to commitment history / idx / accessor segments; the commitment domain snapshot itself is never filtered.
Configured value is persisted in kv.DatabaseInfo (key pruneCommitmentHistory, type older, mirroring how pruneHistory / pruneBlocks are stored). Shrinks of the kept window are allowed; expansions and N → unlimited transitions are rejected because they would require re-downloading filtered files.

Naming aligns with the existing prune namespace --prune.<category>.{older,before} so a future anchored variant (--prune.commitment-history.before=<block>) and other categories slot in without flag-naming churn.

Closes #21198

Out of scope

--prune.commitment-history.before=<block> (anchored "keep history from block N onward") — follow-up PR, but the flag namespace already accommodates it.
Removing already-downloaded files when the bound is tightened — flagged as a follow-up in the issue (state-aggregator reader-lifecycle work).

Test plan

make lint (clean; run twice given non-determinism)
make erigon integration (clean)
go test ./db/kv/prune/... ./db/snapshotsync/... — all passing
- TestEnsureCommitmentHistoryOlderCompatible — first-set / shrink-allowed / expand-rejected / bounded→unlimited rejected / unlimited→bounded allowed (values in blocks: 50k, 100k, 200k)
- TestBlocksToStepDistance — boundary cases for the blocks → steps conversion
- TestCommitmentHistoryMinStep — boundary math for the in-step filter (unchanged)
- TestShouldSkipCommitmentHistorySegment — filter behavior across history/idx/accessor commitment files vs domain/transactions/non-commitment (unchanged)
Smoke-test on a devnet with --prune.include-commitment-history --prune.commitment-history.older=100000

…apshots Adds --prune.commitment-history.mode (archive/recent/medium/custom) and --prune.commitment-history.steps for users running with --prune.include-commitment-history who want to bound how many commitment-history snapshot steps to download. Pre-computes the min step from N + max preverified state step before the download loop and skips commitment history/idx/accessor segments whose end step is at or below that boundary. Domain snapshots are never filtered. Configured value is persisted in kv.DatabaseInfo and shrinks are allowed; expansions (would require re-downloading filtered files) are rejected. Closes #21198

AskAlexSharov

we already have --prune.mode probably don't need one more mode?
steps - it's our internal concept - we don't expose it to users over API/flags.
blocks - are better than txNums/steps - because blocks are linked to time. If user need 1month of history (for Complience) - he can't express it in steps terminology.
Users need express 2 things: "prune history Before BlockNum=N" (for example if i deployed my smart contract at block 22M - they i don't care about deper history), "prune history Older than N blocks" (keep 1 month of recent history).

Examples from reth:

[prune.segments]
receipts           = { distance = N } //  keep the last N blocks (rolling)
receipts           = { before = N }    //  keep everything from block N onward (anchored)

[prune.segments]
sender_recovery    = { distance = 216_000 }
transaction_lookup = { distance = 216_000 }
receipts           = { distance = 216_000 }
account_history    = { distance = 216_000 }
storage_history    = { distance = 216_000 }
bodies_history     = { distance = 216_000 }
merkle_changesets  = { distance = 216_000 }   # this is "commitment history"

Examples from Erigon2:

--prune=hrtc      # prune all four categories
--prune=h         # prune only history; keep receipts, tx-lookup, call-traces fully
--prune=          # empty list = full archive (the default)

--prune.h.older N    # keep last N blocks (rolling distance)
--prune.h.before N   # keep from block N onward (anchored)
--prune.r.older N
--prune.r.before N
--prune.t.older N
--prune.t.before N
--prune.c.older N
--prune.c.before N

Don't need implement all in-1 PR. But cli flags naming/namespaces must be designed with this in mind. We intentionally didn't implement it in Erigon3 - to reduce amount of cli flags and biz-logic. But maybe we can Re-Think it now.

….commitment-history.older (blocks) Per Alex's review on #21200: drop the new "mode"/"steps" namespace and align with existing prune flags (--prune.distance, --prune.distance.blocks). - Replace --prune.commitment-history.mode and --prune.commitment-history.steps with --prune.commitment-history.older=N (units: blocks). Leaves room for a future --prune.commitment-history.before=N anchored variant. - ethconfig.Sync: CommitmentHistoryDistanceSteps -> CommitmentHistoryOlder (blocks). - DB key pruneCommitmentHistoryBlocks -> pruneCommitmentHistory, stored alongside a type marker (kv.PruneTypeOlder), mirroring pruneHistory/pruneBlocks. - SyncSnapshots converts blocks -> steps once per run using olderBlocks * maxStateStep / maxBlock; the existing step-based filter (commitmentHistoryMinStep / shouldSkipCommitmentHistorySegment) is unchanged. - Tests rewritten to operate on blocks; new TestBlocksToStepDistance covers the conversion.

JkLondon · 2026-05-23T17:17:09Z

@AskAlexSharov tried to address ur proposal. Now it's block-oriented + dropped "mode"

Copilot

Pull request overview

This PR introduces a new pruning-related configuration to bound how much historical commitment-history snapshot data is downloaded when running with --prune.include-commitment-history. It adds a block-distance retention flag, converts the block window to an internal “step” cutoff once per run, and skips downloading commitment-history history/idx/accessor segments that fall entirely below that cutoff; it also persists the configured retention in kv.DatabaseInfo and adds tests around the new boundary math and compatibility rules.

Changes:

Add --prune.commitment-history.older=<N blocks> and plumb it through CLI → config → snapshot sync logic.
Implement download-time filtering for old commitment-history history/idx/accessor snapshot segments based on a computed min-step bound.
Persist the retention setting in kv.DatabaseInfo (pruneCommitmentHistory) with compatibility checks and add unit tests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`node/ethconfig/config.go`	Adds `CommitmentHistoryOlder` to sync configuration.
`node/eth/backend.go`	Persists/validates the configured commitment-history retention against DB state.
`node/cli/default_flags.go`	Registers the new CLI flag in the default flag set.
`cmd/utils/flags.go`	Defines `--prune.commitment-history.older` and validates it requires `--prune.include-commitment-history` when non-zero.
`db/snapshotsync/snapshotsync.go`	Computes a min-step bound from block distance and skips old commitment-history segments during snapshot download request construction.
`db/snapshotsync/snapshotsync_test.go`	Adds tests for step-bound math and segment-skip behavior.
`db/kv/tables.go`	Adds `PruneCommitmentHistory` key constant for persistence.
`db/kv/prune/commitment_history.go`	Implements persistence + compatibility rules for the new retention setting.
`db/kv/prune/commitment_history_test.go`	Adds tests for the compatibility rules across configuration transitions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

JkLondon · 2026-06-08T11:26:48Z

+func blocksToStepDistance(olderBlocks, maxStateStep, maxBlock uint64) uint64 {
+	if olderBlocks == 0 || maxStateStep == 0 || maxBlock == 0 {
+		return 0
+	}
+	return olderBlocks * maxStateStep / maxBlock
+}


Fixed in abe92d1. blocksToStepDistance now uses bits.Mul64 to detect overflow and falls back to the full range (which commitmentHistoryMinStep turns into "no filtering") instead of wrapping to a bogus smaller distance. Added an overflow-disables-filter case to TestBlocksToStepDistance.

Note I deliberately kept the olderBlocks > maxBlock path producing a distance >= maxStateStep rather than clamping it here: that case is already handled downstream by commitmentHistoryMinStep (it returns 0 = no filtering when distanceSteps >= maxStateStep), and the existing young-chain-blocks-exceed-maxBlock test locks that behavior down.

JkLondon · 2026-06-08T11:26:49Z

+	case configuredBlocks == 0:
+		return stored, fmt.Errorf(
+			"--prune.commitment-history.older is being expanded from %d block(s) to unlimited; previously filtered files would need to be re-downloaded. delete the chaindata folder to start over",
+			stored)
+	case configuredBlocks > stored:
+		return stored, fmt.Errorf(
+			"--prune.commitment-history.older is being expanded from %d to %d block(s); previously filtered files would need to be re-downloaded. delete the chaindata folder to start over",
+			stored, configuredBlocks)


Good catch — fixed in c957fcc. Added a CommitmentHistoryOlderSet bit, set from ctx.IsSet(...) in the CLI and plumbed through ethconfig into EnsureCommitmentHistoryOlderCompatible(tx, configuredBlocks, configured).

When the flag is absent (configured == false) the persisted value is honored unchanged and nothing is rewritten, so a node that previously set a bound starts fine without re-passing the flag. An explicit --prune.commitment-history.older=0 is still rejected as a bounded -> unlimited transition. Regression tests: unset-honors-stored and unset-first-run-stays-unlimited.

…nd when flag omitted EnsureCommitmentHistoryOlderCompatible treated a missing --prune.commitment-history.older flag (zero default) as an explicit request to go unlimited, so any node that had persisted a non-zero bound failed to start on the next restart unless the flag was passed again. Plumb a "flag was set" bit (CommitmentHistoryOlderSet) from the CLI down to the compatibility check; when the flag is absent, the persisted value is honored unchanged and nothing is rewritten.

…ainst overflow blocksToStepDistance multiplied olderBlocks*maxStateStep before dividing, which could overflow uint64 and silently wrap to a bogus (smaller) distance, skipping the wrong commitment-history segments. Detect overflow with bits.Mul64 and fall back to the full range, which disables filtering via commitmentHistoryMinStep.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

JkLondon · 2026-06-09T21:30:06Z

+	for _, p := range preverified.Items {
+		info, stateFile, ok := snaptype.ParseFileName("", p.Name)
+		if !ok || stateFile {
+			continue
+		}
+		if info.To > maxTo {
+			maxTo = info.To
+		}
+	}


Fixed in 7e721b1. getMaxBlockInSnapshots now scans only EL block segments (headers/bodies/transactions) instead of every non-state file. beaconblocks/blobsidecars parse as non-state with slot-based To values (scaled ×1000), so on chains where slots outrun EL blocks they'd inflate maxBlock, shrink the blocks→steps distance, and over-filter commitment history. It's a no-op on mainnet (EL blocks predate the beacon chain so the beacon To stays below the EL tip), but a real correctness fix for PoS-from-genesis chains. Added TestGetMaxBlockInSnapshots with beacon/blobsidecar segments scaled above the EL tip to lock it down.

JkLondon · 2026-06-09T21:30:17Z

+	if ctx.IsSet(CommitmentHistoryOlderFlag.Name) {
+		older := ctx.Uint64(CommitmentHistoryOlderFlag.Name)
+		if older > 0 && !cfg.KeepExecutionProofs {
+			Fatalf("--%s requires --%s", CommitmentHistoryOlderFlag.Name, KeepExecutionProofsFlag.Name)
+		}
+		cfg.CommitmentHistoryOlder = older
+		cfg.CommitmentHistoryOlderSet = true
+	}


Fixed in 7e721b1. The --prune.include-commitment-history requirement now applies whenever the flag is set at all, not only when older > 0. So --prune.commitment-history.older=0 without the parent flag now fails fast with the documented requirement instead of silently marking the value as set and running the persisted-bound compatibility check for a disabled feature. With commitment history enabled, an explicit =0 still marks it set, so the bounded→unlimited transition stays rejected (round-1 behavior preserved).

JkLondon · 2026-06-09T21:30:19Z

+	if ctx.IsSet(CommitmentHistoryOlderFlag.Name) {
+		older := ctx.Uint64(CommitmentHistoryOlderFlag.Name)
+		if older > 0 && !cfg.KeepExecutionProofs {
+			Fatalf("--%s requires --%s", CommitmentHistoryOlderFlag.Name, KeepExecutionProofsFlag.Name)
+		}
+		cfg.CommitmentHistoryOlder = older
+		cfg.CommitmentHistoryOlderSet = true
+	}


Done in 7e721b1. Added prune.ValidateCommitmentHistoryOlder(mode, older) and call it right after prune.FromCli(...) in applyRemainingEthFlags — where cfg.Prune is populated, as you noted. It rejects older > --prune.distance; an unlimited window (older == 0) or an unlimited/expiry History mode imposes no bound. Regression test TestValidateCommitmentHistoryOlder covers within/equal/exceeds-distance plus the two unlimited cases.

…tment-history.older Three fixes from the Copilot review: - getMaxBlockInSnapshots: restrict the max-block scan to EL block segments (headers/bodies/transactions). beaconblocks/blobsidecars parse as non-state with slot-based To values (scaled x1000); on chains where slots outrun EL blocks they would inflate maxBlock, shrink the blocks->steps distance, and over-filter commitment history. No-op on mainnet (EL blocks predate the beacon chain) but a correctness fix for PoS-from-genesis chains. - SetEthConfig: require --prune.include-commitment-history whenever --prune.commitment-history.older is set at all, not only when >0. An explicit =0 without the parent flag no longer marks the value as "set" (which would run the persisted-bound compatibility check for a disabled feature). Matches the flag's documented requirement. - ValidateCommitmentHistoryOlder: reject older > --prune.distance at startup (issue #21198 proposed scope) since commitment history older than the state-history retention cannot serve eth_getProof. Wired in after prune.FromCli in applyRemainingEthFlags. Unlimited window or unlimited/ expiry History mode imposes no bound.

JkLondon requested review from AskAlexSharov, sudeepdino008 and yperbasis as code owners May 14, 2026 21:25

JkLondon self-assigned this May 14, 2026

AskAlexSharov requested changes May 21, 2026

View reviewed changes

Merge branch 'main' into jklondon/prune-features/skip-old-comm-download

ce51546

JkLondon requested a review from AskAlexSharov May 31, 2026 19:07

yperbasis requested a review from Copilot June 1, 2026 09:56

Copilot started reviewing on behalf of yperbasis June 1, 2026 09:56 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

JkLondon added 2 commits June 8, 2026 13:25

AskAlexSharov requested a review from Copilot June 8, 2026 11:50

Copilot started reviewing on behalf of AskAlexSharov June 8, 2026 11:50 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

yperbasis added this to the 3.6.0 milestone Jun 10, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

db/snapshotsync, kv/prune: skip downloading old commitment-history snapshots#21200

db/snapshotsync, kv/prune: skip downloading old commitment-history snapshots#21200
JkLondon wants to merge 6 commits into
mainfrom
jklondon/prune-features/skip-old-comm-download

JkLondon commented May 14, 2026 •

edited

Loading

Uh oh!

AskAlexSharov left a comment

Uh oh!

JkLondon commented May 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

JkLondon Jun 8, 2026

Uh oh!

JkLondon Jun 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

JkLondon Jun 9, 2026

Uh oh!

JkLondon Jun 9, 2026

Uh oh!

JkLondon Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

JkLondon commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Out of scope

Test plan

Uh oh!

AskAlexSharov left a comment

Choose a reason for hiding this comment

Uh oh!

JkLondon commented May 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

JkLondon Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

JkLondon Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

JkLondon Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

JkLondon Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

JkLondon Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JkLondon commented May 14, 2026 •

edited

Loading