storage: harden replica write backpressure mechanism#25261
storage: harden replica write backpressure mechanism#25261craig[bot] merged 3 commits intocockroachdb:masterfrom
Conversation
5758aba to
13bc4d8
Compare
|
Reviewed 8 of 10 files at r1, 7 of 7 files at r3, 2 of 2 files at r4, 2 of 2 files at r5, 1 of 1 files at r6. pkg/storage/queue.go, line 540 at r5 (raw file):
Do we still need the change in r4 if MaybeAddCallback returns true for ranges in purgatory? pkg/storage/replica_backpressure.go, line 145 at r4 (raw file):
Do these errors make it back to the client? What about the race in the comment? Shouldn't we check whether the range is still large enough to require backpressure? Is it possible that we race with the start of a split? That is, if the range has just gotten large enough to require backpressure but hasn't been added to the split queue yet? Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed. pkg/storage/queue.go, line 540 at r5 (raw file): Previously, bdarnell (Ben Darnell) wrote…
Nope, see below. pkg/storage/replica_backpressure.go, line 145 at r4 (raw file): Previously, bdarnell (Ben Darnell) wrote…
Yes they do, and there's really no good way to avoid the race. It's possible that we race with the start of a split. It's also possible that we race with the completion of a split. Even worse, it's possible that we race with the completion of a split and then the range gets too large again, so it looks like the split never happened. It's even possible that this node isn't even the leaseholder anymore and so the splitQueue rejected the split attempt. This was all concerning me to the point where I was probably going to pull this commit. Conservative backpressure is better than over-aggressive backpressure that may cause client-visible errors. I'm glad that it sounds like you agree. Comments from Reviewable |
Fixes cockroachdb#24974. Fixes cockroachdb#25141. Fixes cockroachdb#25142. This change fixes TestStoreRangeSplitBackpressureWrites and increases the its testing area to include cases where a range is large enough to require backpressure but does not have a split in-flight. Release note: None
Fixes cockroachdb#24215. This change rejects requests to unsplittable ranges that need to be backpressured immediately. It does so by looking at the split queue purgatory and rejecting backpressured writes to any range that is in purgatory. Release note: None
Fixes cockroachdb#25036. This fixes the flakes in the test. They have passed 5 straight times for me. It's also easy to prove that backpressure is necessary for the test to pass, just run the following and it quickly fails: ``` set cluster setting kv.range.backpressure_range_size_multiplier=0; ``` Release note: None
13bc4d8 to
5a609b9
Compare
|
Review status: 0 of 4 files reviewed at latest revision, 2 unresolved discussions. pkg/storage/replica_backpressure.go, line 145 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
PTAL. Comments from Reviewable |
|
Reviewed 1 of 8 files at r7, 2 of 2 files at r8, 1 of 1 files at r9. Comments from Reviewable |
|
TFTR! bors r+ |
25261: storage: harden replica write backpressure mechanism r=nvanbenschoten a=nvanbenschoten Fixes #24215. Fixes #25036. Fixes #24974. Fixes #25141. Fixes #25142. This PR includes a change to harden the replica's write backpressure mechanism. We now reject backpressured write when a range is unsplittable. This is important for placing an upper bound on the size of a range when it consists of just a single row (#24215). The change also fixes a few flaky tests that resulted from backpressure not being restrictive enough. 25539: storage: put an Aborted txn inside a TxnAbortedError r=andreimatei a=andreimatei TxnAbortedErrors have an updated transaction it them. It makes sense for that transaction to be marked as Aborted, instead of Pending. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Build succeeded |
Fixes #24215.
Fixes #25036.
Fixes #24974.
Fixes #25141.
Fixes #25142.
This PR includes a change to harden the replica's write backpressure
mechanism. We now reject backpressured write when a range is unsplittable.
This is important for placing an upper bound on the size of a range when it
consists of just a single row (#24215).
The change also fixes a few flaky tests that resulted from backpressure
not being restrictive enough.