Skip to content

kvcoord: fail-fast when all replicas of a range are unavailable #74503

@tbg

Description

@tbg

With #33007, when a range loses quorum, we will generally have SQL clients experience fail-fast behavior: access to the unavailable range will immediately result in an error, as opposed to hanging indefinitely (as is the case in 21.2 and before). However, when a range has lost all replicas (or if all replicas are unreachable) I believe that DistSender will keep retrying forever:

  • look up descriptor (say r1/1 r1/2 r1/3)
  • try r1/1 (fail)
  • try r1/2 (fail)
  • try r1/3 (fail)
  • hit a SendError here
  • eject descriptor & re-lookup, goto beginning

While we do try to be resilient to network blips, there is probably value in a heuristic where if a request has been attempted twice for each possible replica, it's time to give up.

We will want to return a RangeUnavailableError in this case (similar to #74500) and have similar SQL UX (#74502).

Jira issue: CRDB-12121

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-kv-clientRelating to the KV client and the KV interface.C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)T-kvKV Team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions