Skip to content

fix: Ensure watcher reacts to atomic file updates in Kubernetes mounts#2928

Merged
dadrus merged 2 commits intoreleasefrom
fix/secret_reload_in_k8s
Dec 11, 2025
Merged

fix: Ensure watcher reacts to atomic file updates in Kubernetes mounts#2928
dadrus merged 2 commits intoreleasefrom
fix/secret_reload_in_k8s

Conversation

@dadrus
Copy link
Copy Markdown
Owner

@dadrus dadrus commented Dec 11, 2025

Related issue(s)

closes #2926

Checklist

  • I agree to follow this project's Code of Conduct.
  • I have read, and I am following this repository's Contributing Guidelines.
  • I have read the Security Policy.
  • I have referenced an issue describing the bug/feature request.
  • I have added tests that prove the correctness of my implementation.

Description

The watcher implementation did not account for “atomic” file or directory updates, typically performed via symlink replacement (as used by Kubernetes for secret and config map mounts). This PR adds proper handling for such atomic updates.

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 11, 2025

Codecov Report

❌ Patch coverage is 67.50000% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.14%. Comparing base (5e38f33) to head (4fc4848).
⚠️ Report is 3 commits behind head on release.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
internal/watcher/watcher_impl.go 67.50% 9 Missing and 4 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           release    #2928      +/-   ##
===========================================
- Coverage    91.29%   91.14%   -0.16%     
===========================================
  Files          283      283              
  Lines        10672    10705      +33     
===========================================
+ Hits          9743     9757      +14     
- Misses         677      689      +12     
- Partials       252      259       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dadrus dadrus linked an issue Dec 11, 2025 that may be closed by this pull request
2 tasks
@emsearcy
Copy link
Copy Markdown
Contributor

I can confirm this is working for me in Kubernetes. I rotated it twice to ensure that the second file was being watched correctly, too.

for i in 1 2 3; do
  openssl ecparam -name secp384r1 -genkey -noout -out heimdall-key.pem
  openssl req -new -x509 \
    -subj /CN=heimdall \
    -addext keyUsage=digitalSignature \
    -days 365 \
    -key heimdall-key.pem -out heimdall-cert.pem
  key_id="heimdall-$(date +%Y%m%d)-00${i}"
  (sed -E "s/^(-----BEGIN.*)$/\1\nX-Key-ID: $key_id\n/" heimdall-key.pem; cat heimdall-cert.pem) > heimdall-chain-${i}.pem
  rm heimdall-key.pem heimdall-cert.pem
done
kubectl create secret generic heimdall-certs-ad-hoc -n lfx --from-literal="signer.pem=$(cat heimdall-chain-1.pem)" --save-config
# [ ... restart pods on PR version]
kubectl exec -n lfx lfx-platform-heimdall-789665794b-q8lsc -- /opt/heimdall/heimdall --version
# => heimdall version 4fc48483
curl -s http://lfx-platform-heimdall.lfx.svc.cluster.local:4457/.well-known/jwks | jq '.keys[0].kid'
# => "heimdall-20251211-001"
kubectl get -n lfx secret/heimdall-certs-ad-hoc -o yaml | yq ".data[\"signer.pem\"]=\"$(cat heimdall-chain-2.pem | base64)\"" | k apply -f -
# [... small delay]
curl -s http://lfx-platform-heimdall.lfx.svc.cluster.local:4457/.well-known/jwks | jq '.keys[0].kid'
# => "heimdall-20251211-002"
kubectl get -n lfx secret/heimdall-certs-ad-hoc -o yaml | yq ".data[\"signer.pem\"]=\"$(cat heimdall-chain-3.pem | base64)\"" | k apply -f -
# [... small delay]
curl -s http://lfx-platform-heimdall.lfx.svc.cluster.local:4457/.well-known/jwks | jq '.keys[0].kid'
# => "heimdall-20251211-003"

Logs:

% kubectl logs -n lfx deploy/lfx-platform-heimdall | egrep --line-buffered -i "watch|reload|listener|handling modification"
{"_level_name":"DEBUG","version":"1.1","host":"lfx-platform-heimdall-789665794b-q8lsc","timestamp":1765473632,"level":7,"short_message":"Starting watching config files for changes"}
{"_level_name":"INFO","version":"1.1","host":"lfx-platform-heimdall-789665794b-q8lsc","_file":"/heimdall/cert/signer.pem","timestamp":1765473920,"level":6,"short_message":"Signer key store reloaded"}
{"_level_name":"INFO","version":"1.1","host":"lfx-platform-heimdall-789665794b-q8lsc","_file":"/heimdall/cert/signer.pem","timestamp":1765474074,"level":6,"short_message":"Signer key store reloaded"

@dadrus dadrus merged commit 4800b05 into release Dec 11, 2025
19 checks passed
@dadrus dadrus deleted the fix/secret_reload_in_k8s branch December 11, 2025 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

secrets_reload_enabled and Kubernetes secret mounts

2 participants