ParameterStore should reuse AWS Sessions for multiple requests #1194
Description
Describe the solution you'd like
I would like the Parameter Store provider to reuse a single AWS Session (in a worst case, probably easiest todo) on all requests done for a single ExternalSecret resource.
Best case scenario the controller could just keep refreshing a single authenticated session for the lifetime of the controller.
What is the added value?
Looks like ParameterStore (and most likely Secrets Manager) provider issues a fresh session for each request made to the api effectively doubling the number of requests done to external API (and reducing performance).
Combined with using GetParameter API instead of batch GetParameters it results in a lot of waste and possible performance issues for large scale deployments.
I would expect session reuse to significantly improve the reconciliation times.
Give us examples of the outcome
I am not exactly sure what is wrong with the AWS SDK usage or how could it be improved at this point.
Observations (Constraints, Context, etc):
For the last few days i'm looking into performance issues of ESO on our staging cluster (~30 environments deployed at once, ~30 ExternalSecret objects in each, ~5 Parameters in each secret). I am posting my findings as issues.
After observing "principalid" changing between each request to Parameter Store in AWS CloudTrail Lake, Into verified the provider issues a fresh session for every single GetParameter request.
Notice the AssumeRoleWithWebIdentity.requestParameters.roleSessionName matches the following GetParameter.principalid and each has only a single pair of requests:
CloudTrail entries
[
{
"eventTime": "2022-05-27 07:00:00.000",
"operation": "AssumeRoleWithWebIdentity",
"eventSource": "sts.amazonaws.com",
"principalid": "arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets",
"session_username": "",
"username": "system:serviceaccount:external-secrets:external-secrets",
"requestParameters": "{roleArn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, roleSessionName=1653634800072212330}",
"userIdentity": "{type=WebIdentityUser, principalid=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets, arn=null, accountid=null, accesskeyid=null, username=system:serviceaccount:external-secrets:external-secrets, sessioncontext=null, invokedby=null, identityprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B}"
},
{
"eventTime": "2022-05-27 07:00:00.000",
"operation": "GetParameter",
"eventSource": "ssm.amazonaws.com",
"principalid": "A<REDACTED>7:1653634800072212330",
"session_username": "staging-01-external-secrets",
"username": "",
"requestParameters": "{withDecryption=true, name=/staging/kubernetes/staging/defaults/partners-reporting/secret.snowflake_password}",
"userIdentity": "{type=AssumedRole, principalid=A<REDACTED>7:1653634800072212330, arn=arn:aws:sts::5<REDACTED>8:assumed-role/staging-01-external-secrets/1653634800072212330, accountid=5<REDACTED>8, accesskeyid=ASIAXPRZ43VPEWA5YJHK, username=null, sessioncontext={attributes={creationdate=2022-05-27 07:00:00.000, mfaauthenticated=false}, sessionissuer={type=Role, principalid=A<REDACTED>7, arn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, accountid=5<REDACTED>8, username=staging-01-external-secrets}, webidfederationdata={federatedprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B, attributes=null}, sourceidentity=null, ec2roledelivery=null, ec2issuedinvpc=null}, invokedby=null, identityprovider=null}"
},
{
"eventTime": "2022-05-27 07:00:01.000",
"operation": "GetParameter",
"eventSource": "ssm.amazonaws.com",
"principalid": "A<REDACTED>7:1653634801170591852",
"session_username": "staging-01-external-secrets",
"username": "",
"requestParameters": "{withDecryption=true, name=/staging/kubernetes/staging/defaults/partners-reporting/secret.snowflake_password}",
"userIdentity": "{type=AssumedRole, principalid=A<REDACTED>7:1653634801170591852, arn=arn:aws:sts::5<REDACTED>8:assumed-role/staging-01-external-secrets/1653634801170591852, accountid=5<REDACTED>8, accesskeyid=ASIAXPRZ43VPAH2HJEON, username=null, sessioncontext={attributes={creationdate=2022-05-27 07:00:01.000, mfaauthenticated=false}, sessionissuer={type=Role, principalid=A<REDACTED>7, arn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, accountid=5<REDACTED>8, username=staging-01-external-secrets}, webidfederationdata={federatedprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B, attributes=null}, sourceidentity=null, ec2roledelivery=null, ec2issuedinvpc=null}, invokedby=null, identityprovider=null}"
},
{
"eventTime": "2022-05-27 07:00:01.000",
"operation": "AssumeRoleWithWebIdentity",
"eventSource": "sts.amazonaws.com",
"principalid": "arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets",
"session_username": "",
"username": "system:serviceaccount:external-secrets:external-secrets",
"requestParameters": "{roleArn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, roleSessionName=1653634801170591852}",
"userIdentity": "{type=WebIdentityUser, principalid=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets, arn=null, accountid=null, accesskeyid=null, username=system:serviceaccount:external-secrets:external-secrets, sessioncontext=null, invokedby=null, identityprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B}"
},
{
"eventTime": "2022-05-27 07:00:03.000",
"operation": "AssumeRoleWithWebIdentity",
"eventSource": "sts.amazonaws.com",
"principalid": "arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets",
"session_username": "",
"username": "system:serviceaccount:external-secrets:external-secrets",
"requestParameters": "{roleArn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, roleSessionName=1653634803899903136}",
"userIdentity": "{type=WebIdentityUser, principalid=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets, arn=null, accountid=null, accesskeyid=null, username=system:serviceaccount:external-secrets:external-secrets, sessioncontext=null, invokedby=null, identityprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B}"
},
{
"eventTime": "2022-05-27 07:00:03.000",
"operation": "GetParameter",
"eventSource": "ssm.amazonaws.com",
"principalid": "A<REDACTED>7:1653634803899903136",
"session_username": "staging-01-external-secrets",
"username": "",
"requestParameters": "{withDecryption=true, name=/staging/kubernetes/staging/defaults/partners-reporting/secret.snowflake_password}",
"userIdentity": "{type=AssumedRole, principalid=A<REDACTED>7:1653634803899903136, arn=arn:aws:sts::5<REDACTED>8:assumed-role/staging-01-external-secrets/1653634803899903136, accountid=5<REDACTED>8, accesskeyid=ASIAXPRZ43VPLFMKFSO7, username=null, sessioncontext={attributes={creationdate=2022-05-27 07:00:03.000, mfaauthenticated=false}, sessionissuer={type=Role, principalid=A<REDACTED>7, arn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, accountid=5<REDACTED>8, username=staging-01-external-secrets}, webidfederationdata={federatedprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B, attributes=null}, sourceidentity=null, ec2roledelivery=null, ec2issuedinvpc=null}, invokedby=null, identityprovider=null}"
}
]