-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kverserver: store rebalancer pushes leases back to draining nodes #55440
Copy link
Copy link
Closed
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Description
This is the underlying cause of issues #53760 #53725 #54088 #53811
When a node starts draining, it starts transferring its leases away. The drain process does this until there's no lease left to transfer.
However in the meantime the store rebalancer sees the QPS on the other stores, which are taking on the draining node's load, increase. So it then selects the now-draining node as a good candidate to transfer leases to, because it appears to be low on QPS (because it's draining)
So the leases are transferred back to the draining node.
So the graceful drain never completes.
And the node eventually is killed with active leases.
Which disrupts performance.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.