deletionPolicy: Delete/Merge does not remove or update secret when remote key is deleted #4560
Description
Follow the discussion: https://kubernetes.slack.com/archives/C017BF84G2Y/p1742280242131009
Describe the bug
A question about deletionPolicy. When deletionPolicy is set to Delete/Merge, when the remote key in AWS Secrets Manager is deleted, ExternalSecret will message: could not get secret data from provider reason: SecretSyncedError
And the Secret will not be deleted or updated, which is inconsistent with what is mentioned in the document
https://external-secrets.io/latest/guides/ownership-deletion-policy/
$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.32.2
$ helm list -n external-secrets
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
external-secrets external-secrets 1 2025-03-18 19:50:10.838929 +0800 CST deployed external-secrets-0.14.2 v0.14.2
To Reproduce
1, check init env
$ helm list -n external-secrets
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
external-secrets external-secrets 1 2025-03-18 19:50:10.838929 +0800 CST deployed external-secrets-0.14.2 v0.14.2
$ oc get ss
NAME AGE STATUS CAPABILITIES READY
aws-secrets-manager 9m18s Valid ReadWrite True
2,check AWS secret
$ aws secretsmanager get-secret-value --secret-id Secret-80549
{
"ARN": "*****",
"Name": "Secret-80549",
"VersionId": "*******",
"SecretString": "{\"username\":\"jitli\",\"password\":\"123456\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": "2025-03-18T20:12:29.614000+08:00"
}
3, create es with deletionPolicy: Delete
$ cat externalsecret-delete.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: jitli-secret
spec:
refreshInterval: 1s
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: jitli-secret-be-created
creationPolicy: Owner
deletionPolicy: Delete
data:
- secretKey: jitli-password-from-aws
remoteRef:
key: Secret-80549
property: password
$ oc create -f externalsecret-delete.yaml
externalsecret.external-secrets.io/jitli-secret created
4, get es and secret, SecretSynced
$ oc get es
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY
jitli-secret SecretStore aws-secrets-manager 1s SecretSynced True
$ oc get secret jitli-secret-be-created -o yaml
apiVersion: v1
data:
jitli-password-from-aws: MTIzNDU2
kind: Secret
metadata:
annotations:
reconcile.external-secrets.io/data-hash: e2022936aeabe38383bac61b4b435f74
creationTimestamp: "2025-03-18T12:13:54Z"
labels:
reconcile.external-secrets.io/created-by: 0762fb893119ef8ac573778bd1449dd5
reconcile.external-secrets.io/managed: "true"
name: jitli-secret-be-created
namespace: external-secrets
ownerReferences:
- apiVersion: external-secrets.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: ExternalSecret
name: jitli-secret
uid: 5aa7bbe3-6443-48b8-8c22-fd191f1d6abb
resourceVersion: "65543"
uid: 5609d85e-0990-467a-a434-0bea4cee7a98
type: Opaque
5, delete password in AWS
$ aws secretsmanager get-secret-value --secret-id Secret-80549
{
"ARN": "***",
"Name": "Secret-80549",
"VersionId": "*",
"SecretString": "{\"username\":\"jitli\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": "2025-03-18T20:14:48.807000+08:00"
}
6, get es SecretSyncedError
$ oc get es
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY
jitli-secret SecretStore aws-secrets-manager 1s SecretSyncedError False
secret not be delete
$ oc get secret jitli-secret-be-created -o yaml
apiVersion: v1
data:
jitli-password-from-aws: MTIzNDU2
kind: Secret
pod log
oc logs external-secrets-66b4bc6f64-c4j87
{"level":"error","ts":1742299785.2113593,"msg":"Reconciler error","controller":"externalsecret","controllerGroup":"external-secrets.io","controllerKind":"ExternalSecret","ExternalSecret":{"name":"jitli-secret","namespace":"external-secrets"},"namespace":"external-secrets","name":"jitli-secret","reconcileID":"8857f742-0466-4539-8ce8-96ca1d9f3e1b","error":"error processing spec.data[0] (key: Secret-80549), err: key password does not exist in secret Secret-80549","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249"}
Expected behavior
According to the documentation, the secret should be automatically deleted and no err.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.