Skip to content

spanconfigkvsubscriber: conditionally ignore ProtectionPolicies#77392

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
adityamaru:ignore-pts-records
Mar 8, 2022
Merged

spanconfigkvsubscriber: conditionally ignore ProtectionPolicies#77392
craig[bot] merged 1 commit intocockroachdb:masterfrom
adityamaru:ignore-pts-records

Conversation

@adityamaru
Copy link
Copy Markdown
Contributor

@adityamaru adityamaru commented Mar 4, 2022

This change teaches the GetProtectionTimestamps method in the KVSubscriber
to ignore ProtectionPolicies that were written by a backup, and apply to a span
that has been marked as excluded from backup. This ensures that the ProtectionPolicy
written by a backup does not holdup GC on the span since it will not be exporting its
row data.

Informs: #73536

Release note: None

Release justification: low risk update to new functionality

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@adityamaru adityamaru force-pushed the ignore-pts-records branch 2 times, most recently from 68e6d54 to c0b262c Compare March 5, 2022 00:22
@adityamaru adityamaru marked this pull request as ready for review March 5, 2022 00:22
@adityamaru adityamaru requested a review from a team as a code owner March 5, 2022 00:22
@adityamaru adityamaru requested review from arulajmani and removed request for a team March 5, 2022 00:23
@adityamaru adityamaru force-pushed the ignore-pts-records branch from c0b262c to cb86f71 Compare March 5, 2022 02:30
Copy link
Copy Markdown
Collaborator

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change looks good as is, but IMO an alternative worth considering would be to keep the interface as is and teach the KVSubscriber's implementation to ignore certain protection policies when querying it for protected timestamps. The KVSubscriber already has the SpanConfig and this would ensure the caller doesn't have to join the two fields to filter out certain protection policies.

Reviewed 12 of 13 files at r1, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @adityamaru and @arulajmani)


pkg/kv/kvserver/protectedts/ptcache/cache.go, line 139 at r1 (raw file):

		func(rec *ptpb.Record) (wantMore bool) {
			protectionPolicies = append(protectionPolicies,
				roachpb.ProtectionPolicy{ProtectedTimestamp: rec.Timestamp})

Could you write a comment here explaining why the default value of IgnoreIfExcludedFromBackup is the right thing?


pkg/spanconfig/spanconfig.go, line 410 at r1 (raw file):

// protectedts.Cache.
type ProtectedTSReader interface {
	// GetProtectionPolicies returns all protected timestamps that apply to any

nit: s/all protected timestamps/all protection policies/g


pkg/kv/kvserver/replica_protected_timestamp_test.go, line 63 at r1 (raw file):

				mp.protections = append(mp.protections, manualPTSReaderProtection{
					sp:                 roachpb.Span{Key: keys.MinKey, EndKey: keys.MaxKey},
					protectionPolicies: []roachpb.ProtectionPolicy{{ProtectedTimestamp: ts}},

nit: for all these, you could define yourself a helper that constructs protection policies given a list of timestamps.


pkg/kv/kvserver/protectedts/ptcache/cache_test.go, line 412 at r1 (raw file):

						protections[j].ProtectedTimestamp)
				})
				require.Equal(t, []roachpb.ProtectionPolicy{

nit: Similar to my comment above, might be worth extracting this out in a helper that can give you a list of protection policies given a list of timestamps.


pkg/spanconfig/spanconfigptsreader/adapter_test.go, line 68 at r1 (raw file):

	// Add some records to the cache; ensure they're returned.
	mc.protectionPolicies = append(mc.protectionPolicies,

nit: here as well, if you decide to take my suggestion.

@adityamaru adityamaru force-pushed the ignore-pts-records branch from cb86f71 to f67a7d7 Compare March 6, 2022 15:26
@adityamaru adityamaru changed the title spanconfigptsreader: change method to return ProtectionPolicies spanconfigkvsubscriber: conditionally ignore ProtectionPolicies Mar 6, 2022
@adityamaru adityamaru requested a review from arulajmani March 6, 2022 15:27
@adityamaru adityamaru force-pushed the ignore-pts-records branch from f67a7d7 to f06d5df Compare March 6, 2022 20:04
Copy link
Copy Markdown
Collaborator

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the change, this is much cleaner!

Reviewed 14 of 16 files at r2, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @adityamaru and @arulajmani)


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber_test.go, line 33 at r2 (raw file):

}

func tableSpan(tableID uint32) roachpb.Span {

nit: can we make this a local function to the test?


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber_test.go, line 65 at r2 (raw file):

	sp43Cfg.cfg.ExcludeDataFromBackup = true

	subscriber := &KVSubscriber{}

Can we create a test cluster to get all dependencies and replace this with a call to spanconfigkvsubscriber.New instead?


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber_test.go, line 122 at r2 (raw file):

}

func makeSpanAndSpanConfigWithProtectionPolicies(

nit: can we make this a local function to the test?


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber_test.go, line 143 at r2 (raw file):

	context.Context, bool, ...spanconfig.Update,
) (deleted []spanconfig.Target, added []spanconfig.Record) {
	return nil, nil

Here, and below, let's make these panic?

Copy link
Copy Markdown
Contributor Author

@adityamaru adityamaru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @arulajmani)


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber_test.go, line 65 at r2 (raw file):

Previously, arulajmani (Arul Ajmani) wrote…

Can we create a test cluster to get all dependencies and replace this with a call to spanconfigkvsubscriber.New instead?

just called into spanconfigkvsubscriber.New with everything nil'ed out and that works too.


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber_test.go, line 143 at r2 (raw file):

Previously, arulajmani (Arul Ajmani) wrote…

Here, and below, let's make these panic?

Done.

@adityamaru adityamaru force-pushed the ignore-pts-records branch from f06d5df to a528061 Compare March 8, 2022 05:13
This change teaches the `GetProtectionTimestamps` method in the KVSubscriber
to ignore ProtectionPolicies that were written by a backup, and apply to a span
that has been marked as excluded from backup. This ensures that the ProtectionPolicy
written by a backup does not holdup GC on the span since it will not be exporting its
row data.

Informs: cockroachdb#73536

Release note: None

Release justification: low risk update to new functionality
@adityamaru adityamaru force-pushed the ignore-pts-records branch from a528061 to 36f3550 Compare March 8, 2022 09:34
@adityamaru
Copy link
Copy Markdown
Contributor Author

pretty sure "show cluster setting version" timed out after 2m0s (took 2m0.001s): value differs between local setting ([18 8 8 21 16 2 24 0 32 0]) and KV ([18 9 8 21 16 1 24 0 32 144 9]); try again later is a flake. Thanks for the review on this!

bors r=arulajmani

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 8, 2022

Build succeeded:

@craig craig bot merged commit d384f95 into cockroachdb:master Mar 8, 2022
@adityamaru adityamaru deleted the ignore-pts-records branch March 8, 2022 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants