-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: leaseholder preferences are not enforced when the preferred replica is behind the quorum #38065
Copy link
Copy link
Closed
Labels
A-kv-distributionRelating to rebalancing and leasing.Relating to rebalancing and leasing.S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restartingSevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting
Description
A customer recently an into a problem with Follow the Workload.
Background:
Cluster topology is 3 nodes in 3 separate regions
Most connections are going to Asia (using the right connection string)
Determine this is due to the setup of the cluster and their use case
Admin UI is slow because customer is connecting to it through the global LB
Follow-the-workload didn’t seem to work:
- 80-90% of connections were issued in Asia but the leaseholders did not move there
- The leases did not move as evidenced by the SQL CLI
SHOW RANGE - Localities were enabled.
- N1 (asia-southeast1) received nearly all connections & queries
- System.users and wallets table leases were on n2 (us-central1)
- Alter zone configs with leaseholder preferences return successfully but do not actually alter the zone configuration, and leases do not transfer.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-distributionRelating to rebalancing and leasing.Relating to rebalancing and leasing.S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restartingSevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting