Skip to content

Implement SecretsClient.GetSecrets for (more efficient) batch retrieval of secrets #1193

@nazarewk

Description

@nazarewk

Describe the solution you'd like
I would like AWS Parameter Store provider to retrieve entries in a batch using GetParameters instead of 1 by 1 using GetParameter

What is the added value?
Usually a lot less calls to Parameter Store (1 per ExternalSecret instead of 1 per .spec.data entry) helping with scaling for large parameters sets.

Give us examples of the outcome

I would see moving this loop to a new provider.GetSecrets function

for i, secretRef := range externalSecret.Spec.Data {
secretData, err := providerClient.GetSecret(ctx, secretRef.RemoteRef)
if errors.Is(err, esv1beta1.NoSecretErr) && externalSecret.Spec.Target.DeletionPolicy != esv1beta1.DeletionPolicyRetain {
r.recorder.Event(externalSecret, v1.EventTypeNormal, esv1beta1.ReasonDeleted, fmt.Sprintf("secret does not exist at provider using .data[%d] key=%s", i, secretRef.RemoteRef.Key))
continue
}
if err != nil {
return nil, err
}
providerData[secretRef.SecretKey] = secretData
}

Default implementation would just call provider.GetSecret for each entry, but would allow implementing more efficient batch retrieval functions for providers supporting them.

Observations (Constraints, Context, etc):
As a context I was looking into configuring and improving performance of External Secrets for large (10k+ parameters) Parameter Stores. I stumbled upon each PS request having different userIdentity.principalid in reported CloudTrail Lake results and dig further into it. While i did not find the reason for session per each request I found a possible improvement that i described here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    StaleThis issue/Pull Request is stale and will be automatically closedarea/lifecycleIndicates an issue related to the lifecycle of secretskind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions