Requested by @piyush-singh, @dt and @bdarnell, see the discussion in this review thread.

tldr: we want to more finely categorize settings into whether they are safe to change by the end-user of a cluster or whether they should be restricted to the hosting provider. Possibly, we want that categorization to enable a dynamic re-categorizing of settings without performing a cockroachdb upgrade.

Background and motivation

Today cluster settings assume that there is a single entity that "owns" database administration and is fully responsible for the effects of changing cluster settings. With hosted solutions, there are two entities really: the end-user "owns" the data, schema and a lot of parameters, but the hosting provider "owns" the hosting servers, infrastructure and the mechanisms necessary to guarantee SLAs.

The problem in a hosting env is that one actor changing a setting can have $$ impact on the other actor, in ways undesirable and/or unforeseen. We want to limit that risk by introducing a more clear boundary between what's controllable by the end-user (and has zero impact on the hosting provider) and what's controllable by the hosting provider (and has clear + documented + communicated impact on the end-user).

This goal must be satisfied both for dedicated and multi-tenant clusters. We acknowledge that the technical solutions needed for multi-tenant might be slightly different than for dedicated, however the problem to solve remains the same.

See the linked review issue above for a list of considerations and directions.

Strategy wrt CC

• CC Free tier will run on v20.2 and presumably tenants (end-users) will have no access to cluster settings whatsoever.
• CC shared tier is a different story however. This runs on v21.1 and we wish the Shared tier experience to not be significantly different from that on CC dedicated. This means that the UX of cluster settings must be the same in both environments. However we want to do this safely.

Jira issue: CRDB-3392

Jira issue: CRDB-13261

Epic: CRDB-6671