Skip to content

cp related PRs for "don't query in db, what is in files" #20518

Merged
AskAlexSharov merged 5 commits into
release/3.4from
r34-revert-stale-slot-cp-filerange
Apr 13, 2026
Merged

cp related PRs for "don't query in db, what is in files" #20518
AskAlexSharov merged 5 commits into
release/3.4from
r34-revert-stale-slot-cp-filerange

Conversation

@sudeepdino008

@sudeepdino008 sudeepdino008 commented Apr 13, 2026

Copy link
Copy Markdown
Member

we need it because we're already doing out-of-order prune of domains. So need this protection.

@sudeepdino008 sudeepdino008 marked this pull request as draft April 13, 2026 10:55
@sudeepdino008 sudeepdino008 changed the title [wip] cp cherry picks related PRs for "don't query in db, what is in files" + revert stale slot resurrection Apr 13, 2026
@sudeepdino008 sudeepdino008 changed the title cherry picks related PRs for "don't query in db, what is in files" + revert stale slot resurrection cp related PRs for "don't query in db, what is in files" + revert stale slot resurrection Apr 13, 2026
sudeepdino008 and others added 5 commits April 13, 2026 16:34
Closes #20356

- Adjust `fromTxNum` for the DB iterator in `iterateChangedRecent` to
`max(fromTxNum, files.EndTxNum())`, matching the existing pattern in
`HistoryKeyTxNumRange`
- Prevents reading prunable DB entries that are already covered by
segment files
- Add `TestHistory_IterateChangedRecent_SkipsFileRange`: builds files,
prunes DB entries within the file range, verifies `HistoryRange`
spanning both files and DB still returns correct results
Closes #20358

- Adjust `recentIterateRange` DB query bounds to skip txNums already
covered by frozen files
- For ascending: raise the inclusive lower bound (`from`) to
`filesEndTxNum`
- For descending: raise the exclusive lower bound (`to`) to
`filesEndTxNum-1` (so `value > filesEndTxNum-1` ≡ `value >=
filesEndTxNum`)
- Add `TestInvertedIndex_IdxRange_SkipsFileRange`: builds files for 2
steps, prunes DB entries in the file range, verifies IdxRange still
returns correct results for both asc and desc

## Test plan
- [x] New test `TestInvertedIndex_IdxRange_SkipsFileRange` passes
- [x] All existing `TestInvIndex*` / `TestInvertedIndex*` tests pass
- [x] `make lint` clean
- [x] `make erigon integration` builds
- In `RangeAsOf`, the DB iterator (`HistoryRangeAsOfDB`) now starts from
`max(startTxNum, files.EndTxNum())` instead of `startTxNum` directly
- This avoids reading DB entries within the file range, which are
redundant (served by `HistoryRangeAsOfFiles`) and prunable
- Added `TestRangeAsOf_DBIteratorSkipsFileRange` to verify correctness
after pruning DB entries in the file range

Closes #20357

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
…st (#20444)

Fixes #20355

- Skip DB entries whose step falls within the file range (`step.ToTxNum
< files.EndTxNum()`) in both the initial DB cursor push and the advance
loop — they are never loaded into the heap
- Matches the "don't query DB for data in files" pattern from #20360,
- Add `TestDomain_IteratePrefix_PrefersFilesOverDB` that simulates
partial lexicographic prune and verifies file values win

Follow-up: #20474 applies the same fix to
`DomainLatestIterFile.initCursorMDBX()` and `advanceInFiles()`

- [x] New test passes with fix, fails without it
- [x] `make lint` clean
- [x] `make erigon integration` builds
- Remove unused `step` parameter from the `IteratePrefix` callback
signature (interface, implementations, and all call sites)
- Remove `step` field from `CursorItem` struct (only used by
`debugIteratePrefixLatest`)
- Remove `step` field from `DomainDelPrefix` tombs tuple (was stored but
never read)
- Collapse step decode in DB cursor advancement to a single `endTxNum`
assignment
@sudeepdino008 sudeepdino008 changed the title cp related PRs for "don't query in db, what is in files" + revert stale slot resurrection cp related PRs for "don't query in db, what is in files" Apr 13, 2026
@sudeepdino008 sudeepdino008 force-pushed the r34-revert-stale-slot-cp-filerange branch from 93de384 to 609c08a Compare April 13, 2026 11:06
@sudeepdino008 sudeepdino008 marked this pull request as ready for review April 13, 2026 11:06
@AskAlexSharov AskAlexSharov enabled auto-merge (squash) April 13, 2026 11:15
@AskAlexSharov AskAlexSharov merged commit a605845 into release/3.4 Apr 13, 2026
28 of 30 checks passed
@AskAlexSharov AskAlexSharov deleted the r34-revert-stale-slot-cp-filerange branch April 13, 2026 11:18
AskAlexSharov added a commit that referenced this pull request Apr 18, 2026
we need it because we're already doing out-of-order prune of domains. So
need this protection.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
github-merge-queue Bot pushed a commit that referenced this pull request Apr 20, 2026
Cherry-pick from `release/3.4` to `main`:

- #20635 db/state: fix MaxStep using MinTxNum in DomainRoTx.prune
- #20639 dbg.SaveHeapProfileNearOOMPeriodically() call in all
sub-commands
- #20571 blk-range: wrong logs
- #20565 merge: set merge workers to 1
- #20576 simplify user-visible logs
- #20518 cp related PRs for "don't query in db, what is in files"
- #20517 teach: Unwind beyond data in snapshots not allowed
- #20368 rpc: use synctest for TestClientCancelHTTP to fix CI flakiness
- #20329 logs: move some Info logs to Debug level
- #20262 stageloop, execmodule: add ERIGON_IN_MEM_HISTORY env flag for
in-mem history reads

---------

Co-authored-by: bendertherobert <bendertherobert@gmail.com>
Co-authored-by: moskud <sudeepdino008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: info@weblogix.biz <admin@10gbps.weblogix.it>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants