db: support narrow checkpoints#2237
Conversation
jbowens
left a comment
There was a problem hiding this comment.
Looks great!
Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @RaduBerinde)
checkpoint.go line 50 at r1 (raw file):
// that don't overlap with any of these spans are excluded from the checkpoint. // Note that the checkpoint can still contain keys outside of these spans (from // the WAL and from SSTs that partially overlap with these spans).
nit: maybe add a word of warning that any keys outside the specified spans that do make it into the checkpoint may be inconsistent: iterating over the checkpoint outside the checkpoint spans may surface keys that were deleted or overwritten, etc.
checkpoint.go line 76 at r1 (raw file):
case cmp(f.Smallest.UserKey, s.End) >= 0: // File range starts after range of interest case cmp(f.Largest.UserKey, s.Start) < 0:
a file's largest bound can be exclusive if the file contains range tombstones or range keys that have an exclusive end at f.Largest.UserKey. If f.Largest.IsExclusiveSentinel() and cmp(f.Largest.UserKey, s.Start) == 0, the file f does not contain any keys containing s.Start.
ddb6ac0 to
a2e919a
Compare
This commit adds a checkpoint option to filter out SSTs that don't overlap with a given set of spans. This is useful when we are interested only in a single range (or a few ranges) and don't want to the checkpoint to take up large amounts of space as the store goes through compactions. Fixes cockroachdb#2045
a2e919a to
4576c9d
Compare
RaduBerinde
left a comment
There was a problem hiding this comment.
TFTR!
Reviewable status: 1 of 5 files reviewed, 2 unresolved discussions (waiting on @jbowens)
checkpoint.go line 50 at r1 (raw file):
Previously, jbowens (Jackson Owens) wrote…
nit: maybe add a word of warning that any keys outside the specified spans that do make it into the checkpoint may be inconsistent: iterating over the checkpoint outside the checkpoint spans may surface keys that were deleted or overwritten, etc.
Good point, done.
checkpoint.go line 76 at r1 (raw file):
Previously, jbowens (Jackson Owens) wrote…
a file's largest bound can be exclusive if the file contains range tombstones or range keys that have an exclusive end at
f.Largest.UserKey. Iff.Largest.IsExclusiveSentinel()andcmp(f.Largest.UserKey, s.Start) == 0, the filefdoes not contain any keys containings.Start.
Done, extracted this logic out from some existing code.
jbowens
left a comment
There was a problem hiding this comment.
Reviewed 1 of 1 files at r3, all commit messages.
Reviewable status: 2 of 5 files reviewed, all discussions resolved (waiting on @RaduBerinde)
checkpoint.go line 76 at r1 (raw file):
Previously, RaduBerinde wrote…
Done, extracted this logic out from some existing code.
Nice
This commit adds a checkpoint option to filter out SSTs that don't overlap with a given set of spans. This is useful when we are interested only in a single range (or a few ranges) and don't want to the checkpoint to take up large amounts of space as the store goes through compactions.
Fixes #2045