-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvserver: prototype dropping incoming raft messages based on bytes budget #79752
Copy link
Copy link
Open
Labels
C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-kvKV TeamKV Team
Description
Is your feature request related to a problem? Please describe.
We see in #78408 (comment) and in internal experiments that followers can accrue unbounded amounts of memory in incoming (but not yet processed) raft messages.
Describe the solution you'd like
We are discussing "smarter" solutions in #79215 but in the short term, we need at least a configurable cross-replica memory limit, which, when exhausted, leads to messages getting dropped to protect the recipient node. This limit should be set so that it doesn't fire under normal operations, but will provide a fail-safe. We need to validate that this solves the problem experimentally.
A prototype exists in #79378.
Jira issue: CRDB-15935
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-kvKV TeamKV Team