storage: check ShouldQueue again in baseQueue before requeuing#21673
Merged
nvb merged 1 commit intocockroachdb:masterfrom Jan 23, 2018
Merged
storage: check ShouldQueue again in baseQueue before requeuing#21673nvb merged 1 commit intocockroachdb:masterfrom
nvb merged 1 commit intocockroachdb:masterfrom
Conversation
Member
Contributor
|
Review status: 0 of 2 files reviewed at latest revision, all discussions resolved, some commit checks failed. Comments from Reviewable |
7325f26 to
8a965d6
Compare
The change in cockroachdb#21562 made it possible for queues to process replicas in parallel. In doing so, it changed how `baseQueue` handled the queuing of replicas that were already processing when `MaybeAdd` was called. This was not an issue before because only a single replica could ever be processed at a time, so it was not an issue to add the replica that was currently processing back into the queue. That said, the previous approach could result in replicas being queued with the wrong priority, which could result in queuing inefficiencies. With the changes made in cockroachdb#21562, `baseQueue` now needs to be careful that the same replica doesn't get processed twice at the same time. This is accomplished by using a `requeue` flag on the `replicaItem`. If the flag is set when a replica finishes processing, we requeue it. This ensures that we never skip a necessary processing attempt, and is safe because all queue processing is idempotent. This change makes sure that we call `ShouldQueue` again before placing replicas with the `requeue` flag back in the queue. This isn't strictly required for correctness (again, because processing is idempotent), but it ensure that we don't add replicas back into the queue needlessly. It also ensures that we place the replica back in the queue using the priority provided by `ShouldQueue` *after* the earlier processing attempt finishes, instead of the priority it had *before*. This fixes a subtle issue that we've actually always had. I suspect that this would have helped keep the queue size down in some of the testing from cockroachdb#21562. It would have also helped us in keeping the max range size down because we would have better optimized the ranges that we prioritized when the queue began to fill. Release note (performance improvement): Multiple ranges can now split at the same time, improving our ability to handle hotspot workloads.
8a965d6 to
a2bb38e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The change in #21562 made it possible for queues to process replicas in
parallel. In doing so, it changed how
baseQueuehandled the queuing ofreplicas that were already processing when
MaybeAddwas called. Thiswas not an issue before because only a single replica could ever be
processed at a time, so it was not an issue to add the replica that was
currently processing back into the queue. That said, the previous approach
could result in replicas being queued with the wrong priority, which
could result in queuing inefficiencies.
With the changes made in #21562,
baseQueuenow needs to be careful thatthe same replica doesn't get processed twice at the same time. This is
accomplished by using a
requeueflag on thereplicaItem. If the flagis set when a replica finishes processing, we requeue it. This ensures
that we never skip a necessary processing attempt, and is safe because
all queue processing is idempotent.
This change makes sure that we call
ShouldQueueagain before placingreplicas with the
requeueflag back in the queue. This isn't strictlyrequired for correctness (again, because processing is idempotent), but
it ensure that we don't add replicas back into the queue needlessly. It
also ensures that we place the replica back in the queue using the
priority provided by
ShouldQueueafter the earlier processing attemptfinishes, instead of the priority it had before. This fixes a subtle
issue that we've actually always had.
I suspect that this would have helped keep the queue size down in some
of the testing from #21562. It would have also helped us in keeping
the max range size down because we would have better optimized the
ranges that we prioritized when the queue began to fill.
Release note (performance improvement): Multiple ranges can now split at
the same time, improving our ability to handle hotspot workloads.