[AWS] Support the new EKS Pod Identity Agent #2951
Description
Description of the bug
External Secrets v0.9.9 installed in EKS does not work with the new EKS Pod Identity Agent feature (alternative to IRSA).
EKS Pod Identity Agent injects the AWS_CONTAINER_CREDENTIALS_FULL_URI: http://169.254.170.23/v1/credentials environment variable in the pod, but ESO logs the following error (probably from the AWS SDK):
2023/12/13 09:44:17 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed. <nil>
And thus cannot validate and make ready SecretStores that leverage AWS Secrets Manager.
To Reproduce
Steps to reproduce the behavior:
- On an EKS cluster, version 1.28
- Install the EKS Pod Identity Addon on EKS
aws eks create-addon --cluster-name my-cluster --addon-name eks-pod-identity-agent --addon-version v1.0.0-eksbuild.1 - Create an IAM role for External Secrets with the following policy and trust relationship:
{
"Statement": [
{
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}- Create a pod identity association
aws eks create-pod-identity-association --cluster-name my-cluster --role-arn arn:aws:iam::111122223333:role/my-role --namespace external-secrets --service-account external-secrets - Install ESO with Helm, version 0.9.9 on EKS 1.28, specifying these values:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role
name: external-secrets- Create a ClusterSecretStore (or SecretStore)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: secret-store
spec:
provider:
aws:
region: eu-west-3 # or another region
service: SecretsManagerExpected behavior
The ClusterSecretStore should be ready and valid when I run kubectl get clustersecretstore
NAME AGE STATUS CAPABILITIES READY
secret-store 1m Valid ReadWrite True
Additional outputs/screenshots
The describe output of the external-secrets pod, with the env variables injected by the Pod Identity Agent:
Containers:
external-secrets:
Container ID: containerd://8047ff645c41068b5715ccfe0d9f8d77b4b46fae27d24a0c26680b2bb2170ca7
Image: ghcr.io/external-secrets/external-secrets:v0.9.9
Image ID: ghcr.io/external-secrets/external-secrets@sha256:88d34938c949af80498b87cdbf6fcedf1d10a80190ff126adfcd9db964bfdf65
Port: 8080/TCP
Host Port: 0/TCP
SeccompProfile: RuntimeDefault
Args:
--concurrent=1
State: Running
Started: Wed, 13 Dec 2023 10:43:01 +0100
Ready: True
Restart Count: 0
Limits:
memory: 256Mi
Requests:
cpu: 10m
memory: 32Mi
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: eu-west-3
AWS_REGION: eu-west-3
AWS_CONTAINER_CREDENTIALS_FULL_URI: http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6t6f (ro)
/var/run/secrets/pods.eks.amazonaws.com/serviceaccount from eks-pod-identity-token (ro)
The output of kubectl get clustersecretstores:
NAME AGE STATUS CAPABILITIES READY
secret-store 11m ValidationFailed False
Additional context / potential solution
A fix has been introduced in AWS SDK Go v1.47.11.
ESO v0.9.9 uses AWS SDK v1.47.3 so this explains why the bug is present.
This should be fixed with the next stable release of ESO will be available since AWS SDK has been updated on the main branch.