Skip to content

kverserver: store rebalancer pushes leases back to draining nodes #55440

@knz

Description

@knz

This is the underlying cause of issues #53760 #53725 #54088 #53811

When a node starts draining, it starts transferring its leases away. The drain process does this until there's no lease left to transfer.

However in the meantime the store rebalancer sees the QPS on the other stores, which are taking on the draining node's load, increase. So it then selects the now-draining node as a good candidate to transfer leases to, because it appears to be low on QPS (because it's draining)

So the leases are transferred back to the draining node.

So the graceful drain never completes.
And the node eventually is killed with active leases.
Which disrupts performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions