Skip to content

db: support narrow checkpoints#2237

Merged
RaduBerinde merged 1 commit intocockroachdb:masterfrom
RaduBerinde:narrow-checkpoints
Jan 13, 2023
Merged

db: support narrow checkpoints#2237
RaduBerinde merged 1 commit intocockroachdb:masterfrom
RaduBerinde:narrow-checkpoints

Conversation

@RaduBerinde
Copy link
Copy Markdown
Member

This commit adds a checkpoint option to filter out SSTs that don't overlap with a given set of spans. This is useful when we are interested only in a single range (or a few ranges) and don't want to the checkpoint to take up large amounts of space as the store goes through compactions.

Fixes #2045

@RaduBerinde RaduBerinde requested a review from jbowens January 11, 2023 22:04
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Contributor

@jbowens jbowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Looks great!

Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @RaduBerinde)


checkpoint.go line 50 at r1 (raw file):

// that don't overlap with any of these spans are excluded from the checkpoint.
// Note that the checkpoint can still contain keys outside of these spans (from
// the WAL and from SSTs that partially overlap with these spans).

nit: maybe add a word of warning that any keys outside the specified spans that do make it into the checkpoint may be inconsistent: iterating over the checkpoint outside the checkpoint spans may surface keys that were deleted or overwritten, etc.


checkpoint.go line 76 at r1 (raw file):

		case cmp(f.Smallest.UserKey, s.End) >= 0:
			// File range starts after range of interest
		case cmp(f.Largest.UserKey, s.Start) < 0:

a file's largest bound can be exclusive if the file contains range tombstones or range keys that have an exclusive end at f.Largest.UserKey. If f.Largest.IsExclusiveSentinel() and cmp(f.Largest.UserKey, s.Start) == 0, the file f does not contain any keys containing s.Start.

This commit adds a checkpoint option to filter out SSTs that don't
overlap with a given set of spans. This is useful when we are
interested only in a single range (or a few ranges) and don't want to
the checkpoint to take up large amounts of space as the store goes
through compactions.

Fixes cockroachdb#2045
Copy link
Copy Markdown
Member Author

@RaduBerinde RaduBerinde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

Reviewable status: 1 of 5 files reviewed, 2 unresolved discussions (waiting on @jbowens)


checkpoint.go line 50 at r1 (raw file):

Previously, jbowens (Jackson Owens) wrote…

nit: maybe add a word of warning that any keys outside the specified spans that do make it into the checkpoint may be inconsistent: iterating over the checkpoint outside the checkpoint spans may surface keys that were deleted or overwritten, etc.

Good point, done.


checkpoint.go line 76 at r1 (raw file):

Previously, jbowens (Jackson Owens) wrote…

a file's largest bound can be exclusive if the file contains range tombstones or range keys that have an exclusive end at f.Largest.UserKey. If f.Largest.IsExclusiveSentinel() and cmp(f.Largest.UserKey, s.Start) == 0, the file f does not contain any keys containing s.Start.

Done, extracted this logic out from some existing code.

Copy link
Copy Markdown
Contributor

@jbowens jbowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r3, all commit messages.
Reviewable status: 2 of 5 files reviewed, all discussions resolved (waiting on @RaduBerinde)


checkpoint.go line 76 at r1 (raw file):

Previously, RaduBerinde wrote…

Done, extracted this logic out from some existing code.

Nice

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

db: narrow checkpoints

3 participants