Background information

File based settings create a reserved cluster state, preventing REST mutation of any settings/entities created in this mode. The only way these settings can be modified is through a change in the operator/settings.json file inside our config directory.

When we do snapshot restore we can also bring back the global cluster state, effectively rewriting the cluster state and with it any reserved state. This introduces few issues related to file based settings:

• If there is an operator/settings.json file, it effectively won't be the source of truth anymore, until an update is made to this file.
• If the restore is from another cluster, the metadata version of the reserved state can be higher than the current cluster version accounting, meaning settings updates might fail even if the file is updated.
• If the cluster we are restoring into doesn't use file settings, but the snapshot we are restoring had some reserved state, we would be permanently locking from updates those settings that were reserved.

Adding support for snapshots in file based settings

To support snapshots in file based settings we take the following approach:

1. If the snapshot doesn't restore the global state, we don't do anything.
2. In case we are restoring the global state, we now call the file settings service to:
   • If there's an operator/settings.json file: reset the reserved state metadata version to 0 and touch the file so that the watcher service will apply the settings.
   • If there's no operator/settings.json, we remove the reserved cluster state effectively unlocking those settings that were reserved.

Relates to #89183