Skip to content

kvserver: provide escape hatch for per-replica proposal quota pool #77251

@irfansharif

Description

@irfansharif

Is your feature request related to a problem? Please describe.

We've observed instances where a single node/store that gets into high read-amp regimes is able to affect cluster-wide performance for foreground traffic. Draining these nodes to shed leases can help stave off write pressure and allow compactions to catch up, but it's still possible for follower replicas to cause impact. One avenue for this is our use of proposal quota pools: we maintain one per-range to limit how far ahead a majority quorum can get relative to the slowest replica in the group.

// proposalQuota is the quota pool maintained by the lease holder where
// incoming writes acquire quota from a fixed quota pool before going
// through. If there is no quota available, the write is throttled
// until quota is made available to the pool.
// Acquired quota for a given command is only released when all the
// replicas have persisted the corresponding entry into their logs.
proposalQuota *quotapool.IntPool

The quota pool is helpful in making sure that we're not spending all our time generating snapshots to catch up slow replicas. But when a replica is on a sick node and we're looking to limit its blast radius, it works against us. It's possible then for a slow replica (say on one of these high read-amp nodes) to throttle writes to the corresponding ranges.
#75978 is relevant.

Describe the solution you'd like

It would useful to have the ability to disable the quota pool out-of-band during escalations, and/or control dynamically how far ahead a majority quorum can get before throttling writes. Strawman: We could power this through a hidden cluster setting, or a more targeted crdb_internal built-in to select only specific ranges.

Jira issue: CRDB-13499

Epic CRDB-15069

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-kv-replicationRelating to Raft, consensus, and coordination.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-postmortemOriginated from a Postmortem action item.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions