kvserver: stop reconciling lease count imbalances in the StoreRebalancer#75999
Conversation
|
This change is |
Nodes 1 to 7 are in the region the |
nvb
left a comment
There was a problem hiding this comment.
Reviewed all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @aayushshah15 and @nvanbenschoten)
pkg/kv/kvserver/store_rebalancer.go, line 443 at r1 (raw file):
// Don't bother moving leases whose QPS is below some small fraction of the // store's QPS (unless the store has extra leases to spare anyway). It's
Do we want to remove this part of the comment? And should we have done the same thing in #64559?
Code quote:
(unless the store has extra leases to spare anyway)Release note: None
…ncer` Generally, the `StoreRebalancer` doesn't try to attempt lease transfers for replicas that account for less than 0.1% of the store's total QPS. However, it allows such lease transfers if the store has a higher than average number of leases. This is fraught because of the following reasons: 1. It doesn't adhere to any padding. It will do these tiny lease transfers until the store has less than or equal to the average number of leases. 2. Additionally, "average" here means the average across the cluster, not just the stores that have replicas for the range. This is clearly not good, as this doesn't translate well to heterogeneously loaded clusters. 3. These lease transfers also don't adhere to `kv.allocator.min_lease_transfer_interval`. This means they are not rate limited like the lease transfers done by the replicateQueue. This patch fixes this behavior. This patch is similar to cockroachdb#64559. Noticed while running a tpcc workload on a multi-region cluster, but on a database constrained to a single region. This was done to simulate a "heterogeneously loaded cluster". Release note: None
a490d84 to
5657073
Compare
aayushshah15
left a comment
There was a problem hiding this comment.
TFTRs!
bors r+
Reviewable status:
pkg/kv/kvserver/store_rebalancer.go, line 443 at r1 (raw file):
Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Do we want to remove this part of the comment? And should we have done the same thing in #64559?
Done and done.
|
Build failed (retrying...): |
|
Build succeeded: |
Generally, the
StoreRebalancerdoesn't try to attempt lease transfers forreplicas that account for less than 0.1% of the store's total QPS. However, it
allows such lease transfers if the store has a higher than average number of
leases. This is fraught because of the following reasons:
until the store has less than or equal to the average number of leases.
the stores that have replicas for the range. This is clearly not good, as this
doesn't translate well to heterogeneously loaded clusters.
kv.allocator.min_lease_transfer_interval. This means they are not ratelimited like the lease transfers done by the replicateQueue.
This patch fixes this behavior. This patch is similar to #64559.
Noticed while running a tpcc workload on a multi-region cluster, but on a
database constrained to a single region. This was done to simulate a
"heterogeneously loaded cluster".
Release note: None