Skip to content

iterateChangedRecent: DB query reads data within file range (prunable data) #20356

@sudeepdino008

Description

@sudeepdino008

Parent issue

Part of #16239 — avoiding inconsistency due to lexicographic prune.

Problem

In db/state/history.go, the iterateChangedRecent function creates a HistoryChangesIterDB with startTxKey set from fromTxNum:

if fromTxNum >= 0 {
    binary.BigEndian.PutUint64(s.startTxKey[:], uint64(fromTxNum))
}

When toTxNum extends beyond the file range (or is -1/unlimited), the DB is queried starting from fromTxNum, which may be within the file range. This means DB entries for txNums already covered by files are read and returned. If those DB entries are pruned (in any order), the results would be incomplete.

The rangeIsInFiles early return only handles the case where the entire range is in files. When the range spans both file and DB ranges, the DB portion starts too early.

Fix

Adjust fromTxNum for the DB iterator to max(fromTxNum, files.EndTxNum()):

dbFrom := fromTxNum
if len(ht.iit.files) > 0 {
    dbFrom = max(fromTxNum, int(ht.iit.files.EndTxNum()))
}
if dbFrom >= 0 {
    binary.BigEndian.PutUint64(s.startTxKey[:], uint64(dbFrom))
}

This follows the same pattern already used in HistoryKeyTxNumRange (~line 1503):

dbFrom = max(fromTxNum, int(ht.iit.files.EndTxNum()))

The caller HistoryRange unions the file iterator (iterateKeyTxNumFrozen) with this DB iterator. The file iterator handles [fromTxNum, files.EndTxNum()) and the DB iterator handles [files.EndTxNum(), toTxNum).

Note

PR #17559 contains this fix but has not been merged.

Test needed

Add TestHistory_IterateChangedRecent_SkipsFileRange: write history data across multiple steps, build files, call HistoryRange spanning both file and DB ranges, prune DB entries within the file range, verify results are still correct.

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions