Fix segment file deletion for limited (range-truncated) segrefs#597
Merged
Fix segment file deletion for limited (range-truncated) segrefs#597
Conversation
update_segments/2 used the `--` operator to compute which new segment
files should be deleted as overwritten. `--` performs exact tuple
matching, so when compact_segrefs/2 *limited* (truncated the end index
of) a segment rather than removing it entirely, the limited tuple
{Fn, {Start, NewEnd}} no longer matched the original
{Fn, {Start, OrigEnd}}. The original was then incorrectly included
in OverwrittenSegments, causing the segment file to be deleted while
a reference to it remained in the segment_refs structure.
This manifested when the segment writer sent a batch containing both
normal (4096-entry) segments and compacted (wide-range) segments whose
index ranges overlapped. compact_segrefs would limit the compacted
segments, and the `--` operator would flag them as overwritten. The
files were deleted, leaving dangling references that caused subsequent
reads to fail with "did not found all requested indexes".
Replace the `--` approach with a map-based filename lookup: build a
set of filenames surviving in the compacted result and only flag
segments whose filename is completely absent — i.e. truly removed,
not merely limited.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
update_segments/2 used the
--operator to compute which new segment files should be deleted as overwritten.--performs exact tuple matching, so when compact_segrefs/2 limited (truncated the end index of) a segment rather than removing it entirely, the limited tuple {Fn, {Start, NewEnd}} no longer matched the original {Fn, {Start, OrigEnd}}. The original was then incorrectly included in OverwrittenSegments, causing the segment file to be deleted while a reference to it remained in the segment_refs structure.This manifested when the segment writer sent a batch containing both normal (4096-entry) segments and compacted (wide-range) segments whose index ranges overlapped. compact_segrefs would limit the compacted segments, and the
--operator would flag them as overwritten. The files were deleted, leaving dangling references that caused subsequent reads to fail with "did not found all requested indexes".Replace the
--approach with a map-based filename lookup: build a set of filenames surviving in the compacted result and only flag segments whose filename is completely absent — i.e. truly removed, not merely limited.