Skip to content

kvprober: probe correct keys in crdb_internal.probe_ranges#101554

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/fix101549
Apr 17, 2023
Merged

kvprober: probe correct keys in crdb_internal.probe_ranges#101554
craig[bot] merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/fix101549

Conversation

@nvb
Copy link
Copy Markdown
Contributor

@nvb nvb commented Apr 14, 2023

Fixes #101549.

In #101549, we found that crdb_internal.probe_ranges was unintentionally probing at global range start keys, instead of local probe keys derived from these range start keys. This could lead to serious corruption across the cluster which manifests itself in different ways, depending on the range. This commit fixes crdb_internal.probe_ranges, ensuring that it probes at the correct local probe keys.

The fix also ensures that we don't make this kind of mistake again. It unifies key encoding code paths for the multiple uses of kvprober to avoid bugs. It also adds validation directly above kvprober's access to the KV client which ensures that probe keys are valid.

Release note: None

@nvb nvb requested a review from a team as a code owner April 14, 2023 16:36
@nvb nvb requested a review from a team April 14, 2023 16:36
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@nvb nvb added backport-22.1.x backport-23.1.x PAST MAINTENANCE SUPPORT: 23.1 patch releases via ER request only labels Apr 14, 2023
Fixes cockroachdb#101549.

In cockroachdb#101549, we found that `crdb_internal.probe_ranges` was unintentionally
probing at global range start keys, instead of local probe keys derived from
these range start keys. This could lead to serious corruption across the cluster
which manifests itself in different ways, depending on the range. This commit
fixes `crdb_internal.probe_ranges`, ensuring that it probes at the correct local
probe keys.

The fix also ensures that we don't make this kind of mistake again. It unifies
key encoding code paths for the multiple uses of kvprober to avoid bugs. It also
adds validation directly above kvprober's access to the KV client which ensures
that probe keys are valid.

Release note: None
@nvb nvb force-pushed the nvanbenschoten/fix101549 branch from 08bac66 to b37bdae Compare April 14, 2023 17:11
Copy link
Copy Markdown
Contributor

@erikgrinaker erikgrinaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate the extra assertions and API safety here.

Given the severity, I'd be inclined to get this in for 23.1.0 as well, to avoid anyone accidentally breaking a 23.1.0 cluster because they thought the bug was fixed.

Copy link
Copy Markdown
Collaborator

@nicktrav nicktrav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Thanks for the additional test case too.

@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Apr 17, 2023

TFTRs!

bors r=erikgrinaker,nicktrav

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Apr 17, 2023

Build succeeded:

@blathers-crl
Copy link
Copy Markdown

blathers-crl bot commented Apr 17, 2023

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from b37bdae to blathers/backport-release-22.1-101554: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.1.x failed. See errors above.


error creating merge commit from b37bdae to blathers/backport-release-22.2-101554: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.


error setting reviewers, but backport branch blathers/backport-release-23.1-101554 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/101662/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 23.1.x failed. See errors above.


error setting reviewers, but backport branch blathers/backport-release-23.1.0-101554 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/101663/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 23.1.0 failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@nvb nvb deleted the nvanbenschoten/fix101549 branch April 17, 2023 19:57
@shralex shralex added the O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs label May 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-23.1.x PAST MAINTENANCE SUPPORT: 23.1 patch releases via ER request only O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kvprober: crdb_internal.probe_ranges writes to global keyspace, will corrupt tables

5 participants