-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvserver: drained nodes can re-acquire + retain leases #74691
Copy link
Copy link
Open
Labels
A-kv-decom-rolling-restartDecommission and Rolling RestartsDecommission and Rolling RestartsA-kv-distributionRelating to rebalancing and leasing.Relating to rebalancing and leasing.A-server-start-drainPertains to server startup and shutdown sequencesPertains to server startup and shutdown sequencesC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-server-and-securityDB Server & SecurityDB Server & Security
Description
Describe the problem
In a recent customer escalation we observed a cluster with v21.1.3 binaries where a node that was drained, when re-drained appeared to have re-acquired range leases. The re-drain attempt should've been a no-op; the node was not restarted in the interim.
$ crdb node drain --host :26256 --cluster-name=<redacted>
node is draining... remaining: 129
node is draining... remaining: 0 (complete)
$ crdb node drain --host :26256 --cluster-name=<redacted>
node is draining... remaining: 2
node is draining... remaining: 1
node is draining... remaining: 0 (complete)
Should this be possible?
To Reproduce
Haven't tried reproducing.
Expected behavior
Drained nodes to not acquire additional leases.
+cc @knz for routing/triage.
Jira issue: CRDB-12212
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-decom-rolling-restartDecommission and Rolling RestartsDecommission and Rolling RestartsA-kv-distributionRelating to rebalancing and leasing.Relating to rebalancing and leasing.A-server-start-drainPertains to server startup and shutdown sequencesPertains to server startup and shutdown sequencesC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-server-and-securityDB Server & SecurityDB Server & Security