-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Implement SecretsClient.GetSecrets for (more efficient) batch retrieval of secrets #1193
Description
Describe the solution you'd like
I would like AWS Parameter Store provider to retrieve entries in a batch using GetParameters instead of 1 by 1 using GetParameter
What is the added value?
Usually a lot less calls to Parameter Store (1 per ExternalSecret instead of 1 per .spec.data entry) helping with scaling for large parameters sets.
Give us examples of the outcome
I would see moving this loop to a new provider.GetSecrets function
external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go
Lines 557 to 568 in 6abfbf0
| for i, secretRef := range externalSecret.Spec.Data { | |
| secretData, err := providerClient.GetSecret(ctx, secretRef.RemoteRef) | |
| if errors.Is(err, esv1beta1.NoSecretErr) && externalSecret.Spec.Target.DeletionPolicy != esv1beta1.DeletionPolicyRetain { | |
| r.recorder.Event(externalSecret, v1.EventTypeNormal, esv1beta1.ReasonDeleted, fmt.Sprintf("secret does not exist at provider using .data[%d] key=%s", i, secretRef.RemoteRef.Key)) | |
| continue | |
| } | |
| if err != nil { | |
| return nil, err | |
| } | |
| providerData[secretRef.SecretKey] = secretData | |
| } |
Default implementation would just call
provider.GetSecret for each entry, but would allow implementing more efficient batch retrieval functions for providers supporting them.
Observations (Constraints, Context, etc):
As a context I was looking into configuring and improving performance of External Secrets for large (10k+ parameters) Parameter Stores. I stumbled upon each PS request having different userIdentity.principalid in reported CloudTrail Lake results and dig further into it. While i did not find the reason for session per each request I found a possible improvement that i described here.