Describe the problem

SystemOnly settings are not intended to be read from a tenant process. However, nothing actually disallows a tenant process from reading the in-memory default of a SystemOnly setting and taking actions on it.

The settings implementation includes a safety check for this was intended to be enabled in test builds, but that safety check requires (*Values).SetNonSystemTenant is called at some point during tenant startup. Currently we do not call that method so no accesses to SystemOnly settings from tenants are disallowed.

This recently tripped us up here #91824 and a quick test run with SetNonSystemTenant immediately hit our safety check on scheduler_latency.sample_period. It is likely that there are a number of other SystemOnly settings being accessed from tenant processes.

We should enable SetNonSystemTenant in our tests and work through the problems.

Jira issue: CRDB-21443
Epic: CRDB-6671