Preflight checklist

• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines."

Describe the issue

Based on the recommendations in the Security page section on secrets rotation, I was expecting we could use our external secrets manager, combined with Kubernetes secret mounts, for Heimdall to pull in updated certificate chains for rotating JWT signer keys. ... it didn't work!

More details:

• Used HEIMDALLCFG_SECRETS__RELOAD__ENABLED: true set via Helm chart (under env heading). I see the "Starting watching config files for changes" debug message in the logs.
• Using volumes and volumeMounts to mount a secret with a "signer.pem" attribute to a mountPath of /heimdall/certs, which creates a /heimdall/certs/signer.pem file that is working as expected for the JWT finalizer and JWKS endpoint
• Used "kubectl apply" to update the signer.pem value of the existing secret resource, and confirmed that the container's mounted signer.pem shows the new payload (as expected per k8s docs)
• No "TLS key store reloaded" or "TLS key store reload failed" logs are emitted by the expected OnChanged handler

In researching more, I suspect this is caused by Heimdall checking the fired event with evt.Has(fsnotify.Write) and the peculiar way that Kubernetes does atomic secrets updates using symlinks as described here:

• https://ahmet.im/blog/kubernetes-inotify/
• docs: inotify behavior on atomic configMap/secret volumes is not documented kubernetes/kubernetes#112677

Possible paths forward?

• Updating Security documentation to explain that secrets_reload_enabled doesn't work on Kubernetes. (Users will need to manually cycle pods when secrets change, or leverage an Operator that does this)
• Updating the watcher implementation to handle these events (e.g. IN_DELETE_SELF or however fsnotify/fsnotify abstracts this...), including re-subscribing to the new files. Looking at a couple of linked PRs from the k8s issue, it looks like NATS updated their fsnotify usage to handle this (including a timing delay for the new file to appear), while Thanos moved away from fsnotify entirely to polling. (interesting!)

Version

heimdall version v0.17.5

Additional Context

No response