kv: post-lease transfer promotion from expiration-based lease to epoch-based lease causes failed writes #117630
Description
In fc2d180, we adjusted the lease transfer protocol to pass through an expiration-based lease before being upgraded to an expiration-based lease. This works, but it means that the lease sequence advances twice during a lease transfer. As a result, writes proposed under the expiration-based lease can be re-ordered with the lease promotion and will fail below raft.
This can lead to RETRY_ASYNC_WRITE_FAILURE errors and may be the underlying cause of #90656 and #98553.
We should fix this, ideally by promoting from an expiration-based lease to an epoch-based lease without advancing the lease sequence number.
Jira issue: CRDB-35282
Metadata
Metadata
Assignees
Labels
Anything in KV that doesn't belong in a more specific category.Related to the introduction of Read CommittedCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsIssues/test failures with a fix SLA of 3 monthsKV Team