GCP regional PushSecrets are not working due to replication config #4812
Description
Describe the bug
Related to #4656
ESO has introduced support for GCP regional secrets, but there is currently a bug in PushSecrets.
When using the PushSecrets feature with a GCP secret store configured for regional secrets, we encounter the following error related to the replication configuration:
Error Message Example
status:
conditions:
- lastTransitionTime: "2025-04-14T13:49:13Z"
message: 'set secret failed: could not write remote ref password to target secretstore
gcp-store: rpc error: code = InvalidArgument desc = Secret.replication
should not be provided.'
reason: Errored
status: "False"
type: Ready
syncedPushSecrets:
ClusterSecretStore/gcp-store: {}PR #4751 has not fixed the root cause of the problem because by default the replication is set to Automatic, but it should not be set at all when using Regional Secrets.
See the official comparison between global and regional secrets: https://cloud.google.com/secret-manager/docs/secret-manager-secrets-comparison
To Reproduce
- Environment Setup:
- Use a Kubernetes cluster with Workload Identity configured.
- Ensure that GCP Secret Manager contains a secret with the corresponding key (e.g., your-secret-key).
- Install ExternalSecrets v0.17.0
- Deploy the ExternalSecrets controller using version v0.17.0.
- Deploy ClusterSecretStore:
Use a configuration similar to the following (with internal names replaced) (the service account has the role roles/secretmanager.admin):
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: <your-clustersecretstore-name>
spec:
provider:
gcpsm:
auth:
workloadIdentity:
clusterLocation: <your-cluster-location>
clusterName: <your-cluster-name>
serviceAccountRef:
name: <your-serviceaccount-name>
namespace: <your-serviceaccount-namespace>
location: <your-cluster-location>
projectID: <your-gcp-project-id>- Deploy following resources:
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
name: test-pwgen
spec:
allowRepeat: true
digits: 5
length: 32
noUpper: false
symbolCharacters: -_
symbols: 5
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: test-password
spec:
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
name: test-pwgen
refreshInterval: 0s
target:
creationPolicy: Owner
deletionPolicy: Retain
name: test-password-generated
---
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: test-secret
spec:
data:
- conversionStrategy: None
match:
remoteRef:
remoteKey: test-secret
secretKey: password
metadata:
apiVersion: kubernetes.external-secrets.io/v1alpha1
kind: PushSecretMetadata
spec:
labels:
managed-by: external-secrets
deletionPolicy: None
refreshInterval: 1h
secretStoreRefs:
- kind: ClusterSecretStore
name: <your-clustersecretstore-name>
selector:
secret:
name: test-password-generated
updatePolicy: Replace- Get the error message
$ kubectl describe pushsecrets.external-secrets.io test-secret
(...)
Warning Errored 45s (x5 over 98s) pushsecret set secret failed: could not write remote ref password to target secretstore gcp-store: rpc error: code = InvalidArgument desc = Secret.replication should not be provided.Expected behavior
-
For Global Secrets:
- Continue to support user-managed replication and automatic replication
-
For Regional Secrets:
- Do not set and send the replication field at all
Additional context
Synchronising regional secrets works fine, only using PushSecrets does not work.