Problem

When there's an ephemeral encryption key for the encrypted saved-object attributes plugin, as soon as the Kibana process restarts the encryption key is lost and anything that was encrypted with it can't be decrypted. Alerting is currently the primary consumer of encrypted saved-object attributes, as such I've focused on the impact this has on Alerting for a large portion of the problem statement. We have other ephemeral encryption keys in Kibana, but their impact is a lot less severe given their usage patterns.

If a user was to create an Alert with an ephemeral encryption key, and then the Kibana process restarts, the Alert would just stop working. To mitigate a user creating a bunch of Alerts which won't work as soon as the Kibana process restarts, Alerting blocks the user from creating Alerts until an encryption key is set in the kibana.yml.

Regardless of the encryption key being ephemeral or set in the kibana.yml, it should be the same value for all Kibana instances. Otherwise, in a high-availability deployment of Kibana with multiple instances, if an Alert is created by an instance of Kibana that has a different encryption key than the instance that tries to run it, the alert will fail:

Potential Solutions

Please feel free to recommend others besides the following, they're almost certainly not exhaustive.

When there is an ephemeral encryption key, warn the user and let them store the secrets unencrypted in plain-text

This works "out of the box" for high availability deployments of Kibana. The secrets are stored in plain-text in the .kibana index, and as such users would be able to read them. When the user later sets a proper encryption key, we'd have to encrypt all of the existing plain-text secrets.

Use a default encryption key and warn the user when they're using the default

This also works "out of the box" for high availability deployments of Kibana. The secrets aren't stored in plain-text in the .kibana index, but since they're using a well-known encryption key it's trivial for an attacker to decrypt them. This is honestly more obfuscation than real security. When the user later sets a proper encryption key, we'd have to decrypt all of the existing encrypted secrets using the default encryption key and use the new encryption key.

Generate a random encryption key on the first start-up and persist it to the kibana.yml or the keystore

This doesn't work "out of the box" with high availability deployments. It also is potentially problematic for situations like Docker where there isn't a persistent filesystem. It's secure because no users with access to the .kibana index are able to decrypt the values. At the moment, our linux packages (.deb/.rpm) allow the Kibana process to write to either the kibana.yml or the keystore, but we haven't utilized this approach elsewhere and ideally we could restrict write access to both of these to prevent an attacker from overwriting configuration values.

Create a CLI tool to walk users through configuration Kibana's various encryption keys

Users at the moment have to manually know the 3 settings which must be set to properly configure encryption for security, reporting and encrypted saved-object attributes. If we offered the user a CLI tool similar to Elasticsearch's cert-util we could interactively walk them through setting up Kibana, including the configuration of the encryption keys.