Skip to content

kvserver: load based rebalancing within cold regions in heterogenous workloads #69901

@aayushshah15

Description

@aayushshah15

With #65379, the StoreRebalancer is able to perform QPS-based lease and replica rebalancing in the "hot" regions in a cluster experiencing heterogeneous load across its regions.

However, as of the time of writing this, the StoreRebalancer is only activated on stores serving QPS sufficiently above the average QPS across all stores in the cluster. This means that, in a heterogeneously loaded cluster, the colder regions won't rebalance within themselves until the degree of imbalance gets high enough that one of the stores is above the "overfullness threshold".

The caveat against lifting this restriction is that we might be too aggressive in performing load based {lease,replica} rebalancing on stores that have no need for it, and end up actually hurting QPS overall due to the overheads associated with lease transfers and replica rebalances.

We should investigate the impact of allowing load based rebalances on colder stores, and / or predicate the activation of the StoreRebalancer on a tunable QPS threshold.

Jira issue: CRDB-9853

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-kv-distributionRelating to rebalancing and leasing.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)X-staleno-issue-activity

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions