This is a call for feedback. We would like to find out what the use cases for transient settings are, and whether we can address them in other ways instead, allowing us to remove the transient cluster settings feature and simplify our story around settings (and precedence order).

While Elasticsearch's configurability is very powerful, it can also lead to confusion and sometimes misconfigurations. Typical configuration options include:

• regular node settings (not updateable, added in elasticsearch.yml or as command-line argument when starting up Elasticsearch)
• secure node settings (dynamically reloadable, part of keystore, uses CLI for updating)
• transient and persistent cluster settings (updated through REST API), broadcasted through the cluster state and applied on all nodes in cluster
• consistent settings (secure node setting values that need to be equal on all nodes)
• settings can be filtered (e.g. to mask password/credentials)
• settings are layered (transient cluster setting taking precedence over persistent cluster setting over node setting ...)

We would like to simplify the story on how to configure Elasticsearch. The proposal here is to remove transient cluster settings. These are rarely used, often cause confusion when used in combination with persistent cluster settings, and make our APIs unnecessarily complex.

Transient cluster settings

• have an unpredictable lifecycle. In contrast to persistent cluster settings, they do not survive across full cluster restarts. The problem with this definition is that full cluster restart is a bit ill-defined here. It does not necessarily mean that the user has explicitly restarted all nodes, but also if enough master-eligible nodes crash, the cluster has to recover the cluster state from disk, which misses the transient settings, amounting to the effect of a full cluster restart. This means that the loss of transient settings can be quite unexpected and is not necessarily tied to a user-controlled action (restarting all nodes).
• make our APIs unnecessarily complex. When configuring Elasticsearch, users have to concern themselves with whether the setting should be a node setting, a cluster setting, whether it should be transient or persistent, worry about setting precedences etc. In most cases, persistent cluster settings should be the way to go. Removing one choice here will make it easier for our users to do the right thing.

Please raise any scenarios that you feel can currently only be solved by transient settings, so that we can look into alternative ways to address them.