Skip to content

increase ChangeSets3 prune limit at chain tip#21204

Merged
sudeepdino008 merged 2 commits into
performancefrom
mdbx_bloat_changesets3_prune
May 15, 2026
Merged

increase ChangeSets3 prune limit at chain tip#21204
sudeepdino008 merged 2 commits into
performancefrom
mdbx_bloat_changesets3_prune

Conversation

@sudeepdino008

@sudeepdino008 sudeepdino008 commented May 15, 2026

Copy link
Copy Markdown
Member

Summary

On heavy-state chains (bloatnet), ChangeSets3 was the dominant chaindata growth source post-catch-up — file grew unboundedly because prune couldn't keep up with the per-block changeset write rate.

Root cause: the pruneDiffsLimitOnChainTip = 1000 cap in PruneExecutionStage (active when initialCycle=false). On bloatnet:

  • per-block changeset entries: ~1000–1500 (each ~5 KB serialized diff chunks)
  • per commit-cycle: ~40 blocks executed → ~40k–60k entries written
  • per commit-cycle: ChangeSets3 prune drains at most 1000 (or until 2s timeout) → drain rate is roughly 1–2% of write rate
  • net: ChangeSets3 grows ~1–2 GB per minute under heavy load, pushing chaindata file size up by tens of GB per hour

Observed on a 12-hour bloatnet run: ChangeSets3 stayed at 0 B during catch-up (initialCycle=true overrides the cap to math.MaxInt), then ballooned from 0 → 40 GB in the ~3 hours after the chain caught up. File size grew 38 GB → 181 GB over the same window, with ~80% of the new space attributable to ChangeSets3 + write amplification from a too-small reclaim pool.

Changes

  1. execution/stagedsync: bump ChangeSets3 chain-tip prune limit 1000 → 200000.
    The 2s timeout still bounds wall time; the cap raise removes the artificial floor on how many entries one call drains. With 200k cap × 2s timeout, a single PruneExecutionStage invocation can drain up to ~1 GB of changesets — well above the per-cycle write rate.

  2. db/rawdb: PruneTable: fold logEvery + ctx + timeout into one mod-1000 check.
    Per-iteration select-on-logEvery.C was a syscall on every row. Moved into the same mod-stride as ctx-done + timeout, and bumped stride 100 → 1000. For 200k-row prunes this shaves the per-iter overhead noticeably without affecting timeout responsiveness (1000 iters at ~microseconds each = under 10 ms granularity).

Notes

  • Catch-up path (initialCycle=true) is unaffected — the override there already uses math.MaxInt / 1h.
  • Mainnet's per-block changeset rate is much lower than bloatnet's, so the old 1000 cap was rarely binding. The new 200k cap is just as benign there (the 2s timeout caps actual work).
  • The bump pairs with the prune-in-CommitCycle change (execution/execmodule: prune chaindata in CommitCycle (+ release roTx, bloatnet collation gate) #21192) — that gave us a second prune call per FCU iteration, but both paths shared the 1000 cap. Doubling calls doesn't help if each is throttled.

Test plan

  • CI on `performance`
  • Mainnet sync still healthy (cap raise + stride change are non-functional w.r.t. correctness; only affect drain throughput)

…00000

On heavy-state chains (bloatnet) the 1000-entry per-call cap drains
~85x slower than ChangeSets3's creation rate once initialCycle=false.
The 2s timeout still bounds wall-time; 200k just removes the artificial
floor on how many entries one call can touch.
… check

Per-iteration select-on-logEvery was a syscall on every row. Moved
into the same mod-stride as ctx-done + timeout, and bumped stride
from 100 to 1000. For 200k-row prunes this shaves the
per-iter overhead noticeably without affecting timeout responsiveness
(1000 iters at \~microseconds each = under 10ms granularity).
@sudeepdino008 sudeepdino008 marked this pull request as draft May 15, 2026 01:56
@sudeepdino008 sudeepdino008 changed the title execution/stagedsync, db/rawdb: speed up ChangeSets3 prune at chain tip [wip] speed up ChangeSets3 prune at chain tip May 15, 2026
@sudeepdino008 sudeepdino008 changed the title [wip] speed up ChangeSets3 prune at chain tip increase ChangeSets3 prune limit at chain tip May 15, 2026
@sudeepdino008 sudeepdino008 marked this pull request as ready for review May 15, 2026 10:21
@sudeepdino008 sudeepdino008 enabled auto-merge (squash) May 15, 2026 10:21
@sudeepdino008 sudeepdino008 merged commit 31382d1 into performance May 15, 2026
69 checks passed
@sudeepdino008 sudeepdino008 deleted the mdbx_bloat_changesets3_prune branch May 15, 2026 10:56
ADD12 pushed a commit to ADD12/erigon that referenced this pull request May 16, 2026
pull Bot pushed a commit to Dustin4444/erigon that referenced this pull request May 29, 2026
Cherry-pick of 5 performance improvements from the `performance` branch
to `main`:

1. warmuper: cancelable worker — erigontech#20941
2. db/state, cmd/integration: 4x larger commitment rebuild shard,
squeeze flag transparent — erigontech#21147
3. db/rawdb: increase ChangeSets3 prune loop stride 100→1000, move log
inside stride check — erigontech#21204
4. db/seg: increase bufio pool size from 256KB to 512KB
5. db/kv/prune: remove dead `limit` parameter from `TableScanningPrune`
(accepted but never forwarded internally)

---------

Co-authored-by: awskii <awskii@users.noreply.github.com>
Co-authored-by: moskud <sudeepdino008@gmail.com>
Co-authored-by: info@weblogix.biz <admin@10gbps.weblogix.it>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants