Skip to content

storage: harden replica write backpressure mechanism#25261

Merged
craig[bot] merged 3 commits intocockroachdb:masterfrom
nvb:nvanbenschoten/splitQueueBP
May 16, 2018
Merged

storage: harden replica write backpressure mechanism#25261
craig[bot] merged 3 commits intocockroachdb:masterfrom
nvb:nvanbenschoten/splitQueueBP

Conversation

@nvb
Copy link
Copy Markdown
Contributor

@nvb nvb commented May 2, 2018

Fixes #24215.
Fixes #25036.
Fixes #24974.
Fixes #25141.
Fixes #25142.

This PR includes a change to harden the replica's write backpressure
mechanism. We now reject backpressured write when a range is unsplittable.
This is important for placing an upper bound on the size of a range when it
consists of just a single row (#24215).

The change also fixes a few flaky tests that resulted from backpressure
not being restrictive enough.

@nvb nvb requested review from a team, m-schneider and tbg May 2, 2018 23:38
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@bdarnell
Copy link
Copy Markdown
Contributor

bdarnell commented May 3, 2018

Reviewed 8 of 10 files at r1, 7 of 7 files at r3, 2 of 2 files at r4, 2 of 2 files at r5, 1 of 1 files at r6.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.


pkg/storage/queue.go, line 540 at r5 (raw file):

	if purgatoryErr, ok := bq.mu.purgatory[rangeID]; ok {
		cb(purgatoryErr)
		return true

Do we still need the change in r4 if MaybeAddCallback returns true for ranges in purgatory?


pkg/storage/replica_backpressure.go, line 145 at r4 (raw file):

		}) {
			// No split ongoing. We may have raced with its completion.
			return errors.Errorf("range large enough to require backpressure, but no split ongoing")

Do these errors make it back to the client?

What about the race in the comment? Shouldn't we check whether the range is still large enough to require backpressure?

Is it possible that we race with the start of a split? That is, if the range has just gotten large enough to require backpressure but hasn't been added to the split queue yet?


Comments from Reviewable

@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented May 16, 2018

Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


pkg/storage/queue.go, line 540 at r5 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

Do we still need the change in r4 if MaybeAddCallback returns true for ranges in purgatory?

Nope, see below.


pkg/storage/replica_backpressure.go, line 145 at r4 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

Do these errors make it back to the client?

What about the race in the comment? Shouldn't we check whether the range is still large enough to require backpressure?

Is it possible that we race with the start of a split? That is, if the range has just gotten large enough to require backpressure but hasn't been added to the split queue yet?

Yes they do, and there's really no good way to avoid the race. It's possible that we race with the start of a split. It's also possible that we race with the completion of a split. Even worse, it's possible that we race with the completion of a split and then the range gets too large again, so it looks like the split never happened. It's even possible that this node isn't even the leaseholder anymore and so the splitQueue rejected the split attempt. This was all concerning me to the point where I was probably going to pull this commit. Conservative backpressure is better than over-aggressive backpressure that may cause client-visible errors. I'm glad that it sounds like you agree.


Comments from Reviewable

nvb added 3 commits May 16, 2018 14:49
Fixes cockroachdb#24974.
Fixes cockroachdb#25141.
Fixes cockroachdb#25142.

This change fixes TestStoreRangeSplitBackpressureWrites and
increases the its testing area to include cases where a range
is large enough to require backpressure but does not have a
split in-flight.

Release note: None
Fixes cockroachdb#24215.

This change rejects requests to unsplittable ranges that need to be
backpressured immediately. It does so by looking at the split queue
purgatory and rejecting backpressured writes to any range that is
in purgatory.

Release note: None
Fixes cockroachdb#25036.

This fixes the flakes in the test. They have passed 5 straight times for
me. It's also easy to prove that backpressure is necessary for the test
to pass, just run the following and it quickly fails:

```
set cluster setting kv.range.backpressure_range_size_multiplier=0;
```

Release note: None
@nvb nvb force-pushed the nvanbenschoten/splitQueueBP branch from 13bc4d8 to 5a609b9 Compare May 16, 2018 18:53
@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented May 16, 2018

Review status: 0 of 4 files reviewed at latest revision, 2 unresolved discussions.


pkg/storage/replica_backpressure.go, line 145 at r4 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Yes they do, and there's really no good way to avoid the race. It's possible that we race with the start of a split. It's also possible that we race with the completion of a split. Even worse, it's possible that we race with the completion of a split and then the range gets too large again, so it looks like the split never happened. It's even possible that this node isn't even the leaseholder anymore and so the splitQueue rejected the split attempt. This was all concerning me to the point where I was probably going to pull this commit. Conservative backpressure is better than over-aggressive backpressure that may cause client-visible errors. I'm glad that it sounds like you agree.

PTAL.


Comments from Reviewable

@bdarnell
Copy link
Copy Markdown
Contributor

:lgtm:


Reviewed 1 of 8 files at r7, 2 of 2 files at r8, 1 of 1 files at r9.
Review status: all files reviewed at latest revision, all discussions resolved.


Comments from Reviewable

@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented May 16, 2018

TFTR!

bors r+

craig bot pushed a commit that referenced this pull request May 16, 2018
25261: storage: harden replica write backpressure mechanism r=nvanbenschoten a=nvanbenschoten

Fixes #24215.
Fixes #25036.
Fixes #24974.
Fixes #25141.
Fixes #25142.

This PR includes a change to harden the replica's write backpressure
mechanism. We now reject backpressured write when a range is unsplittable.
This is important for placing an upper bound on the size of a range when it
consists of just a single row (#24215).

The change also fixes a few flaky tests that resulted from backpressure
not being restrictive enough.

25539: storage: put an Aborted txn inside a TxnAbortedError r=andreimatei a=andreimatei

TxnAbortedErrors have an updated transaction it them. It makes sense for
that transaction to be marked as Aborted, instead of Pending.

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
@craig
Copy link
Copy Markdown
Contributor

craig bot commented May 16, 2018

Build succeeded

@craig craig bot merged commit 5a609b9 into cockroachdb:master May 16, 2018
@nvb nvb deleted the nvanbenschoten/splitQueueBP branch May 21, 2018 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants