-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvserver: throttle writes on followers #79215
Copy link
Copy link
Closed
Labels
A-admission-controlA-kv-distributionRelating to rebalancing and leasing.Relating to rebalancing and leasing.A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-sreFor issues SRE opened or otherwise cares about tracking.For issues SRE opened or otherwise cares about tracking.
Metadata
Metadata
Assignees
Labels
A-admission-controlA-kv-distributionRelating to rebalancing and leasing.Relating to rebalancing and leasing.A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-sreFor issues SRE opened or otherwise cares about tracking.For issues SRE opened or otherwise cares about tracking.
We currently throttle writes on the receiving store based on store health (e.g. via admission control or via specialized
AddSSTablethrottling). However, this only takes into account the local store health, and not the associated write cost on followers during replication, which isn't always throttled. We've seen this lead to hotspots where follower stores get overwhelmed, since the follower writes bypass admission control. A similar problem exists with snapshot application.This has been touched on in several other issues as well:
Jira issue: CRDB-14642
Epic CRDB-15069