-
Notifications
You must be signed in to change notification settings - Fork 4.1k
server: avoid starting tenant servers until all overrides have been received #96512
Description
Describe the problem
Currently the SQL server for a secondary tenant can start serving traffic before it has received all its cluster setting overrides.
This is undesirable for a few reasons:
- it can see tenant-all overrides before tenant-specific overrides.
- it makes the behavior non-deterministic
- it makes it impossible to make the server initialization dependent on a cluster setting.
Expected behavior
Look at (c *connector) Start() in kv/kvclient/kvtenant/connector.go. It starts the setting receiver (c.runTenantSettingsSubscription()) but it does not wait for the initial settings to have been received (it only waits for the first, not all of them). So some code after the call to .Start() completes can still see the wrong/outdated values.
The proper fix in combination to your PR is to add a synchronization step there.
The tenant connector should wait until a sentinel value has been received that indicates that all the initial values have been received.
Jira issue: CRDB-24159
Epic: CRDB-26691