Skip to content

storage: leaseholder preferences are not enforced when the preferred replica is behind the quorum #38065

@awoods187

Description

@awoods187

A customer recently an into a problem with Follow the Workload.

Background:
Cluster topology is 3 nodes in 3 separate regions
Most connections are going to Asia (using the right connection string)
Determine this is due to the setup of the cluster and their use case
Admin UI is slow because customer is connecting to it through the global LB

Follow-the-workload didn’t seem to work:

  • 80-90% of connections were issued in Asia but the leaseholders did not move there
  • The leases did not move as evidenced by the SQL CLI SHOW RANGE
  • Localities were enabled.
  • N1 (asia-southeast1) received nearly all connections & queries
  • System.users and wallets table leases were on n2 (us-central1)
  • Alter zone configs with leaseholder preferences return successfully but do not actually alter the zone configuration, and leases do not transfer.

Metadata

Metadata

Assignees

Labels

A-kv-distributionRelating to rebalancing and leasing.S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions