Skip to content

kvserver: drained nodes can re-acquire + retain leases #74691

@irfansharif

Description

@irfansharif

Describe the problem

In a recent customer escalation we observed a cluster with v21.1.3 binaries where a node that was drained, when re-drained appeared to have re-acquired range leases. The re-drain attempt should've been a no-op; the node was not restarted in the interim.

$ crdb node drain --host :26256 --cluster-name=<redacted>
node is draining... remaining: 129
node is draining... remaining: 0 (complete)

$ crdb node drain --host :26256 --cluster-name=<redacted>
node is draining... remaining: 2
node is draining... remaining: 1
node is draining... remaining: 0 (complete)

Should this be possible?

To Reproduce

Haven't tried reproducing.

Expected behavior

Drained nodes to not acquire additional leases.


+cc @knz for routing/triage.

Jira issue: CRDB-12212

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-kv-decom-rolling-restartDecommission and Rolling RestartsA-kv-distributionRelating to rebalancing and leasing.A-server-start-drainPertains to server startup and shutdown sequencesC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-server-and-securityDB Server & Security

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions