Skip to content

kvserver: improve the accounting of AddSSTableRequests' QPS  #73731

@aayushshah15

Description

@aayushshah15

Currently, our QPS accounting is a little broken because we only record the count of incoming BatchRequests into a replica (not the actual count of requests in those batches). One of the consequences of this is that AddSSTTableRequests only count as 1 QPS even though their actual load on the receiving hardware is orders of magnitude higher than that. This means that during an import, without any background traffic in the cluster, CRDB's StoreRebalancer (which is responsible for QPS based lease and replica rebalancing) doesn't even kick in (because it doesn't try resolving tiny QPS differences between stores). We've seen at least one support case where this all lead to severe load imbalance among the nodes in a customer's cluster.

One idea by @nvanbenschoten was to consider AddSSTableRequests to be something like 1000 requests when recording them in the leaseholder replica's leaseholderStats.

This is likely a small change but we'd want to run a few experiments to validate it (same as #50620).

cc. @cockroachdb/kv-notifications

Metadata

Metadata

Assignees

Labels

A-kv-distributionRelating to rebalancing and leasing.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-kvKV Team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions