Skip to content

server: avoid starting tenant servers until all overrides have been received #96512

@knz

Description

@knz

Describe the problem

Currently the SQL server for a secondary tenant can start serving traffic before it has received all its cluster setting overrides.
This is undesirable for a few reasons:

  • it can see tenant-all overrides before tenant-specific overrides.
  • it makes the behavior non-deterministic
  • it makes it impossible to make the server initialization dependent on a cluster setting.

Expected behavior

Look at (c *connector) Start() in kv/kvclient/kvtenant/connector.go. It starts the setting receiver (c.runTenantSettingsSubscription()) but it does not wait for the initial settings to have been received (it only waits for the first, not all of them). So some code after the call to .Start() completes can still see the wrong/outdated values.

The proper fix in combination to your PR is to add a synchronization step there.

The tenant connector should wait until a sentinel value has been received that indicates that all the initial values have been received.

Jira issue: CRDB-24159
Epic: CRDB-26691

Metadata

Metadata

Assignees

Labels

A-multitenancyRelated to multi-tenancyA-server-architectureRelates to the internal APIs and src org for server codeC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-multitenantIssues owned by the multi-tenant virtual teamv23.1.5

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions