We have some settings that it only makes sense to set for the host cluster, but that SQL code still needs to be able to read. There's currently no way to do this, and I see indications that a lot of code thinks it does but actually doesn't. We have three settings:

• SystemOnly: can only be set and read by the host cluster.
• TenantReadOnly: tenants can read the setting, but the setting must be set individually for each tenant. The host cluster setting is invisible, the tenant will see the default setting unless an individual setting is set.
• TenantWritable: the setting is tenant-specific.

Consider e.g. kv.closed_timestamp.side_transport_interval which controls how often we send closed timestamp updates from leaseholders to followers. This is currently TenantWritable, which doesn't make any sense because the side transport doesn't run in a tenant, it runs below KV. However, SQL code (in this case changefeeds) still needs to access this setting, for example because it needs to know how often to expect closed timestamps to be emitted. Currently, they won't see the KV setting, they'll see the tenant setting, which can lead to changefeeds breaking completely in tenants with certain configurations. But if I change it to SystemOnly the setting isn't visible at all (well, code can read it because of a bug, but they'll only see the default value not the host value), and if I change it to TenantReadOnly it will only read the tenant's version of the setting, not the host cluster.

This seems pretty broken, and it isn't clear to me how to proceed here. Do we need a new class, e.g. TenantVisible, or alternatively make SystemOnly visible to tenants (at least from code)?

See also Slack thread.

This has tripped us up in #91824 and #108678, and also applies to e.g. kv.rangefeed.enabled, follower_read_timestamp(), and many other settings.

Jira issue: CRDB-30584
Epic: CRDB-6671