-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvserver: provide escape hatch for per-replica proposal quota pool #77251
Description
Is your feature request related to a problem? Please describe.
We've observed instances where a single node/store that gets into high read-amp regimes is able to affect cluster-wide performance for foreground traffic. Draining these nodes to shed leases can help stave off write pressure and allow compactions to catch up, but it's still possible for follower replicas to cause impact. One avenue for this is our use of proposal quota pools: we maintain one per-range to limit how far ahead a majority quorum can get relative to the slowest replica in the group.
cockroach/pkg/kv/kvserver/replica.go
Lines 568 to 574 in f9dee66
| // proposalQuota is the quota pool maintained by the lease holder where | |
| // incoming writes acquire quota from a fixed quota pool before going | |
| // through. If there is no quota available, the write is throttled | |
| // until quota is made available to the pool. | |
| // Acquired quota for a given command is only released when all the | |
| // replicas have persisted the corresponding entry into their logs. | |
| proposalQuota *quotapool.IntPool |
The quota pool is helpful in making sure that we're not spending all our time generating snapshots to catch up slow replicas. But when a replica is on a sick node and we're looking to limit its blast radius, it works against us. It's possible then for a slow replica (say on one of these high read-amp nodes) to throttle writes to the corresponding ranges.
#75978 is relevant.
Describe the solution you'd like
It would useful to have the ability to disable the quota pool out-of-band during escalations, and/or control dynamically how far ahead a majority quorum can get before throttling writes. Strawman: We could power this through a hidden cluster setting, or a more targeted crdb_internal built-in to select only specific ranges.
Jira issue: CRDB-13499
Epic CRDB-15069