feat(pruner): persist PrunedTxSet across sessions + make cap configurable#873
Closed
freemans13 wants to merge 2 commits into
Closed
feat(pruner): persist PrunedTxSet across sessions + make cap configurable#873freemans13 wants to merge 2 commits into
freemans13 wants to merge 2 commits into
Conversation
…able Two changes that together let the PrunedTxSet optimisation actually bite for the tight-chain workload it was designed for. 1. Persist the set on the Service struct so children whose parents were pruned in earlier sessions can still skip the parent-update round-trip. In the previous design the set was allocated per `PruneWithPartitions` call. With `pruner_block_trigger=OnBlockMined` every block triggers a new session, so a child at height H+1 was never able to recognise its parent at height H — by then the parent's session had ended and the set had been thrown away. On dev-scale-1 this limited the catch rate to ~6.5% of would-be-wasted parent updates (24M / 371M). 2. Replace the hard-coded 10M cap with a `pruner_utxoPrunedSetMaxEntries` setting (default 10M, 0 = unlimited). Persistent sessions accumulate entries across many blocks, so the cap is more likely to saturate; operators tuning for high-fan-out workloads can raise it. Adds two gauge metrics so operators can see what's happening: - utxo_pruner_pruned_set_size — current Len() of the set - utxo_pruner_pruned_set_saturated — 1 if cap reached, 0 otherwise `CheckAndRemove` remains destructive: for parents with high fan-out only the first child to look skips the update. That's a separate concern from persistence and not addressed here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
🤖 Claude Code Review Status: Complete Current Review:
The change enables the optimization to work across block boundaries as intended, with proper memory bounds and monitoring. |
|
Contributor
Benchmark Comparison ReportBaseline: Current: Summary
All benchmark results (sec/op)
Threshold: >10% with p < 0.05 | Generated: 2026-05-18 13:49 UTC |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes that together let the PrunedTxSet optimisation from #628 actually bite for the tight-chain workload it was designed for.
Persist the set across sessions. The set is now owned by the
Servicestruct instead of being allocated perPruneWithPartitionscall. Withpruner_block_trigger=OnBlockMined, every block triggers a new prune session, so a child at height H+1 was never able to recognise its parent at height H — by the time the child's session ran, the parent's session had ended and the set had been thrown away. On dev-scale-1 (image12b6408adf...) this limited the catch rate to ~6.5% of would-be-wasted parent updates (24M / 371M wasteful updates caught). Persistence lets the common cross-block-boundary case fire.Make the cap configurable. Replace the hard-coded
prunedTxSetMaxEntries = 10_000_000const with apruner_utxoPrunedSetMaxEntriessetting (default 10M,0= unlimited). Persistent sessions accumulate entries across many blocks, so the cap is more likely to saturate; operators tuning for high-throughput / high-fan-out workloads can raise it without rebuilding.Observability
Two new gauge metrics:
utxo_pruner_pruned_set_size— currentLen()of the setutxo_pruner_pruned_set_saturated—1if the cap is reached,0otherwiseUse these together with
utxo_pruner_parents_skipped_pruned_totalto confirm whether the cap needs raising.Out of scope
CheckAndRemoveis still destructive — for parents with high fan-out only the first child to look skips the update. That's a separate concern from persistence; if it turns out to matter in production we can swap toContains+ a proper eviction strategy in a follow-up.Test plan
go test ./stores/utxo/aerospike/pruner/... ./settings/...go vetcleanutxo_pruner_parents_skipped_pruned_totaland the new size/saturated gauges; expect the catch rate to climb materially over multiple sessionspruner_utxoPrunedSetMaxEntriesupward and re-measure🤖 Generated with Claude Code