Skip to content

kvserver: prototype dropping incoming raft messages based on bytes budget #79752

@tbg

Description

@tbg

Is your feature request related to a problem? Please describe.

We see in #78408 (comment) and in internal experiments that followers can accrue unbounded amounts of memory in incoming (but not yet processed) raft messages.

Describe the solution you'd like

We are discussing "smarter" solutions in #79215 but in the short term, we need at least a configurable cross-replica memory limit, which, when exhausted, leads to messages getting dropped to protect the recipient node. This limit should be set so that it doesn't fire under normal operations, but will provide a fail-safe. We need to validate that this solves the problem experimentally.

A prototype exists in #79378.

Jira issue: CRDB-15935

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-kvKV Team

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions