Skip to content

cp: integrity check PRs to performance (#20435, #20680, #20714, #20835)#20838

Merged
wmitsuda merged 5 commits into
performancefrom
cherry-pick/20835-to-performance
Apr 27, 2026
Merged

cp: integrity check PRs to performance (#20435, #20680, #20714, #20835)#20838
wmitsuda merged 5 commits into
performancefrom
cherry-pick/20835-to-performance

Conversation

@sudeepdino008

@sudeepdino008 sudeepdino008 commented Apr 27, 2026

Copy link
Copy Markdown
Member

Cherry-picks the integrity-area chain from main onto performance, in order:

  1. integrity: restore StateProgress check #20435integrity: restore StateProgress check
  2. integrity, stagedsync: cap state collation at block snapshots progress #20680integrity, stagedsync: cap state collation at block snapshots progress
  3. add time budget for integrity checks #20714add time budget for integrity checks
  4. cmd/utils/app: skip commitment history integrity checks when disabled #20835cmd/utils/app: skip commitment history integrity checks when disabled

#20714 is the structural refactor of doIntegrity (errgroup.Go(switch{...})runCheck := func{} + budget loop). Conflicted with performance-only Bloatnet customizations on the same function (#20806, #20810, #20813); resolved by adopting #20714's shape and re-applying the Bloatnet handling for CommitmentKvDeref, CommitmentHistVal, and StateRootVerifyByHistory. failFast arg on CheckCommitmentHistAtBlkRange preserved (matches performance's signature).

Submodule pointer for execution/tests/execution-spec-tests left unchanged.

make erigon + make lint clean.

@sudeepdino008 sudeepdino008 marked this pull request as draft April 27, 2026 10:03
sudeepdino008 and others added 4 commits April 27, 2026 12:05
- Reverts #19738
- Restores the check that state/commitment files aren't ahead of block
files

---------

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Add `--integrity.budget <duration>` to `erigon seg integrity` that caps
total wall-clock time for an integrity run.

- When set, checks are sorted cheap → heavy (based on
`integrity.FastChecks` order) and each receives a per-check deadline of
`remaining / remaining_checks`, so finishing early on a cheap check
rolls budget forward to the heavy tail.
- On per-check timeout: log `[integrity] budget exhausted, moving on` at
INFO and continue to the next check. Not treated as an integrity failure
— `--failFast` only reacts to actual integrity errors.
- When not set (default): no change in behavior — user-supplied order
preserved, no deadline applied.
- `BlockReader.IntegrityTxnID` now takes `ctx` and polls every 1000
segments so `BlocksTxnID` honors the budget.
- `CheckKvi` producer now checks `ctx` every 100k keys, so skip-heavy
runs (e.g. `CommitmentKvi` with `SampleRatio=0`) don't overshoot their
slice by minutes.
- `doPublishable` takes `datadir.Dirs` directly instead of
`*cli.Context`.

follow up:
- pick to release/3.4
- change automation script to allot 30min budget

---------

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
…#20835)

## Summary

`seg integrity` runs `CommitmentHistVal` and `StateRootVerifyByHistory`
regardless of whether commitment history is persisted on the datadir.
Both depend on commitment history `.v` files:

- `CheckCommitmentHistVal` iterates `.v` files; when none exist it
returns nil after logging `files=0`, which silently masks
misconfiguration as a passing check.
- `CheckCommitmentHistAtBlkRange` would fail confusingly when trying to
read history that isn't there.

This PR reads `rawdb.ReadDBCommitmentHistoryEnabled` once at the start
of `doIntegrity` and skips both checks with an informative log line when
the flag is off. Mirrors the existing `BorEvents` / `BorSpans`
chain-guard pattern in the same dispatch switch.

The other commitment checks (`CommitmentRoot`, `CommitmentKvi`,
`CommitmentKvDeref`, `StateVerify`) operate on the domain `.kv` files
and don't depend on history, so they remain unconditional.

## Test plan

- [x] `make lint` clean
- [x] `go build ./cmd/utils/app/...` succeeds
- [x] On a no-history datadir, run `seg integrity --check
CommitmentHistVal` and confirm the skip log appears and the command
exits 0 instead of "passing" with files=0
- [x] On a history-enabled datadir, confirm `CommitmentHistVal` and
`StateRootVerifyByHistory` still execute as before
@sudeepdino008 sudeepdino008 force-pushed the cherry-pick/20835-to-performance branch from 057bd81 to a5e0c36 Compare April 27, 2026 10:12
@sudeepdino008 sudeepdino008 changed the title cmd/utils/app: skip commitment history integrity checks when disabled (cp #20835) cmd/utils/app, db/integrity, execution/stagedsync: cp integrity stack to performance (#20435, #20680, #20714, #20835) Apr 27, 2026
@sudeepdino008 sudeepdino008 changed the title cmd/utils/app, db/integrity, execution/stagedsync: cp integrity stack to performance (#20435, #20680, #20714, #20835) cp: integrity check PRs to performance (#20435, #20680, #20714, #20835) Apr 27, 2026
@sudeepdino008 sudeepdino008 marked this pull request as ready for review April 27, 2026 10:38
@sudeepdino008 sudeepdino008 requested a review from wmitsuda April 27, 2026 10:38
@wmitsuda wmitsuda merged commit 0c22920 into performance Apr 27, 2026
32 checks passed
@wmitsuda wmitsuda deleted the cherry-pick/20835-to-performance branch April 27, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants