Workload Identity with AWS & IMDSv2 use expired token #1755
Description
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Environment details
- OS: Linux
- Node.js version: 16.20.2 (I know it's EOL..)
- npm version: 8.19.4
google-auth-libraryversion: 9.4.0
Steps to reproduce
- Setup federation with AWS according to https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds . The JSON file should have
imdsv2_session_token_url. - Observe that API calls to Google (we're testing with service account signJwt) are working normally
- Wait about an hour
- Observe that API calls are now failing
The error object we receive is:
GaxiosError
at Gaxios._request (/app/node_modules/gaxios/build/src/gaxios.js:142:23)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async AwsClient.getAwsRoleName (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:217:26)
at async AwsRequestSigner.getCredentials (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:110:34)
at async AwsRequestSigner.getRequestOptions (/app/node_modules/google-auth-library/build/src/auth/awsrequestsigner.js:66:40)
at async AwsClient.retrieveSubjectToken (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:125:25)
at async AwsClient.refreshAccessTokenAsync (/app/node_modules/google-auth-library/build/src/auth/baseexternalclient.js:288:30)
at async AwsClient.getAccessToken (/app/node_modules/google-auth-library/build/src/auth/baseexternalclient.js:165:13)
[scrubbed] {
config: {
url: 'http://169.254.169.254/latest/meta-data/iam/security-credentials',
method: 'GET',
responseType: 'text',
headers: {
'x-aws-ec2-metadata-token': '<private>',
'User-Agent': 'google-api-nodejs-client/9.6.1',
'x-goog-api-client': 'gl-node/16.20.2'
},
paramsSerializer: [Function: paramsSerializer],
validateStatus: [Function: validateStatus],
errorRedactor: [Function: defaultErrorRedactor]
},
response: {
config: {
url: 'http://169.254.169.254/latest/meta-data/iam/security-credentials',
method: 'GET',
responseType: 'text',
headers: [Object],
paramsSerializer: [Function: paramsSerializer],
validateStatus: [Function: validateStatus],
errorRedactor: [Function: defaultErrorRedactor]
},
data: '',
headers: {
connection: 'close',
'content-length': '0',
'content-type': 'text/plain',
date: 'Wed, 07 Feb 2024 05:38:23 GMT',
server: 'EC2ws'
},
status: 401,
statusText: 'Unauthorized',
request: {
responseURL: 'http://169.254.169.254/latest/meta-data/iam/security-credentials'
}
},
error: undefined,
status: 401,
[Symbol(gaxios-gaxios-error)]: '6.2.0'
}
I think this is because in
google-auth-library-nodejs/src/auth/awsclient.ts
Lines 151 to 161 in 3b19e9c
this.awsRequestSigner is null, then is reused for all subsequent requests.
However in getImdsV2SessionToken the x-aws-ec2-metadata-token-ttl-seconds is only configured to be 300s:
so after the application run for some times the GCP token expires and refreshing of the token fails because it use expired IMDSv2 token.
Semi-related but I checked how the Go SDK has implemented and it seems to be 1:1 match to this behavior and may also have similar problem https://cs.opensource.google/go/x/oauth2/+/refs/tags/v0.17.0:google/internal/externalaccount/aws.go;l=304