Skip to content

storage: check ShouldQueue again in baseQueue before requeuing#21673

Merged
nvb merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/queueReq
Jan 23, 2018
Merged

storage: check ShouldQueue again in baseQueue before requeuing#21673
nvb merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/queueReq

Conversation

@nvb
Copy link
Copy Markdown
Contributor

@nvb nvb commented Jan 22, 2018

The change in #21562 made it possible for queues to process replicas in
parallel. In doing so, it changed how baseQueue handled the queuing of
replicas that were already processing when MaybeAdd was called. This
was not an issue before because only a single replica could ever be
processed at a time, so it was not an issue to add the replica that was
currently processing back into the queue. That said, the previous approach
could result in replicas being queued with the wrong priority, which
could result in queuing inefficiencies.

With the changes made in #21562, baseQueue now needs to be careful that
the same replica doesn't get processed twice at the same time. This is
accomplished by using a requeue flag on the replicaItem. If the flag
is set when a replica finishes processing, we requeue it. This ensures
that we never skip a necessary processing attempt, and is safe because
all queue processing is idempotent.

This change makes sure that we call ShouldQueue again before placing
replicas with the requeue flag back in the queue. This isn't strictly
required for correctness (again, because processing is idempotent), but
it ensure that we don't add replicas back into the queue needlessly. It
also ensures that we place the replica back in the queue using the
priority provided by ShouldQueue after the earlier processing attempt
finishes, instead of the priority it had before. This fixes a subtle
issue that we've actually always had.

I suspect that this would have helped keep the queue size down in some
of the testing from #21562. It would have also helped us in keeping
the max range size down because we would have better optimized the
ranges that we prioritized when the queue began to fill.

Release note (performance improvement): Multiple ranges can now split at
the same time, improving our ability to handle hotspot workloads.

@nvb nvb requested review from a team and bdarnell January 22, 2018 19:30
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@bdarnell
Copy link
Copy Markdown
Contributor

:lgtm:


Review status: 0 of 2 files reviewed at latest revision, all discussions resolved, some commit checks failed.


Comments from Reviewable

@nvb nvb force-pushed the nvanbenschoten/queueReq branch from 7325f26 to 8a965d6 Compare January 23, 2018 15:43
The change in cockroachdb#21562 made it possible for queues to process replicas in
parallel. In doing so, it changed how `baseQueue` handled the queuing of
replicas that were already processing when `MaybeAdd` was called. This
was not an issue before because only a single replica could ever be
processed at a time, so it was not an issue to add the replica that was
currently processing back into the queue. That said, the previous approach
could result in replicas being queued with the wrong priority, which
could result in queuing inefficiencies.

With the changes made in cockroachdb#21562, `baseQueue` now needs to be careful that
the same replica doesn't get processed twice at the same time. This is
accomplished by using a `requeue` flag on the `replicaItem`. If the flag
is set when a replica finishes processing, we requeue it. This ensures
that we never skip a necessary processing attempt, and is safe because
all queue processing is idempotent.

This change makes sure that we call `ShouldQueue` again before placing
replicas with the `requeue` flag back in the queue. This isn't strictly
required for correctness (again, because processing is idempotent), but
it ensure that we don't add replicas back into the queue needlessly. It
also ensures that we place the replica back in the queue using the
priority provided by `ShouldQueue` *after* the earlier processing attempt
finishes, instead of the priority it had *before*. This fixes a subtle
issue that we've actually always had.

I suspect that this would have helped keep the queue size down in some
of the testing from cockroachdb#21562. It would have also helped us in keeping
the max range size down because we would have better optimized the
ranges that we prioritized when the queue began to fill.

Release note (performance improvement): Multiple ranges can now split at
the same time, improving our ability to handle hotspot workloads.
@nvb nvb force-pushed the nvanbenschoten/queueReq branch from 8a965d6 to a2bb38e Compare January 23, 2018 18:18
@nvb nvb merged commit 5e35295 into cockroachdb:master Jan 23, 2018
@nvb nvb deleted the nvanbenschoten/queueReq branch January 23, 2018 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants