Skip to content

kvserver: deflake TestProtectedTimestamps#110072

Merged
craig[bot] merged 2 commits intocockroachdb:masterfrom
arulajmani:fix-109244
Sep 6, 2023
Merged

kvserver: deflake TestProtectedTimestamps#110072
craig[bot] merged 2 commits intocockroachdb:masterfrom
arulajmani:fix-109244

Conversation

@arulajmani
Copy link
Copy Markdown
Collaborator

Previously, we weren't waiting for the PTS record to make its way to KV in this test. This meant expecting it to provide protection (which the test asserted) was racy with spanconfig reconciliation. Span config reconciliation is built over rangefeeds, so when we bumped the default value of kv.rangefeed.closed_timestamp_refresh_interval in #108667, it meant span config reconciliation would take longer. This caused the GC queue to run before the replica was aware of the PTS record, which violated a test assertion.

This patch fixes the issue by only enqueueing the range in the GC queue once the PTS record is known to have made it to KV.

Closes #109244

Release note: None

Previously, we weren't waiting for the PTS record to make its way to KV
in this test. This meant expecting it to provide protection (which the
test asserted) was racy with spanconfig reconciliation. Span config
reconciliation is built over rangefeeds, so when we bumped the default
value of `kv.rangefeed.closed_timestamp_refresh_interval` in cockroachdb#108667,
it meant span config reconciliation would take longer. This caused the
GC queue to run before the replica was aware of the PTS record, which
violated a test assertion.

This patch fixes the issue by only enqueueing the range in the GC queue
once the PTS record is known to have made it to KV.

Closes cockroachdb#109244

Release note: None
@arulajmani arulajmani requested a review from a team as a code owner September 5, 2023 23:28
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

ptsReader := tc.GetFirstStoreFromServer(t, 0).GetStoreConfig().ProtectedTimestampReader
require.NoError(
t,
ptutil.TestingVerifyProtectionTimestampExistsOnSpans(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you rename this helper to something that captures that there's a SucceedsSoon block internally? WaitUntil...? There's one usage that wraps this helper around a SucceedsSoon block as well, so it's confused at least one other person.

TestingWaitForProtectedTimestampToExistOnSpans captures the fact
that there's a SucceedsSoon in there better. While here, we also clean
up a few usages and no longer return a useless error.

Epic: none

Release note: None
Copy link
Copy Markdown
Collaborator Author

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

bors r=irfansharif

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @irfansharif and @pavelkalinnikov)


pkg/kv/kvserver/client_protectedts_test.go line 188 at r1 (raw file):

Previously, irfansharif (irfan sharif) wrote…

Could you rename this helper to something that captures that there's a SucceedsSoon block internally? WaitUntil...? There's one usage that wraps this helper around a SucceedsSoon block as well, so it's confused at least one other person.

Good call, tacked on a commit.

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 6, 2023

Build succeeded:

@craig craig bot merged commit d1ea771 into cockroachdb:master Sep 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kv/kvserver: TestProtectedTimestamps failed

3 participants