(rds): secret rotation application times out before rotation completes #17265
Description
What is the problem?
The secret rotation application times out before rotation completes and rotation fails.
Reproduction Steps
// create DB with generated master DB secret
const db = new rds.DatabaseInstance(this, 'Database', {
engine: rds.DatabaseInstanceEngine.postgres({
version: rds.PostgresEngineVersion.VER_13_4,
}),
instanceType: ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE4_GRAVITON, ec2.InstanceSize.MICRO),
vpc,
});
// add rotation to DB master secret
db.addRotationSingleUser({
automaticallyAfter: cdk.Duration.days(30),
});
// create secondary DB secret (used by apps to connect to DB)
const dbUserSecret = new rds.DatabaseSecret(this, 'DbUserSecret', {
username: 'app_user',
masterSecret: dbAdminUserSecret,
});
// attach secondary DB secret
const dbUserSecretAttached = dbUserSecret.attach(db);
// add rotation to secondary DB secret
new secretsmanager.SecretRotation(this, 'DatabaseUserSecretRotation', {
application: secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
secret: db.secret!,
target: db,
vpc,
automaticallyAfter: cdk.Duration.days(30),
});I log into the DB using the master user postgres and the generated password and create the secondary user app_user with password secret_passwd.
I'm now able to log into the DB using the user app_user with password secret_passwd.
Next, I trigger secret rotation for the secondary DB secret:
aws secretsmanager rotate-secret --secret-id <SECONDARY_DB_SECRET_ID>Alternatively: under AWS Console > AWS Secrets Manager > Secrets > DbUserSecretXXXXXXXX-yyyyyyyyyyyy, press Rotate secret immediately
What did you expect to happen?
After triggering secret rotation, I can log into the DB using the newly generated password and the old password no longer works.
What actually happened?
Secret rotation is triggered successfully, however, I'm not able to login with the newly generated password. The old password secret_passwd still works.
Looking at the CloudWatch Logs of the secret rotation application, I can see that the function repeatedly times out:
2021-11-01T17:29:04.505+01:00 START RequestId: 428afaa9-e114-43b2-8054-b71239dfb8b5 Version: $LATEST
2021-11-01T17:29:04.770+01:00 [INFO] 2021-11-01T16:29:04.769Z 428afaa9-e114-43b2-8054-b71239dfb8b5 Found credentials in environment variables.
2021-11-01T17:34:04.611+01:00 2021-11-01T16:34:04.611Z 428afaa9-e114-43b2-8054-b71239dfb8b5 Task timed out after 300.10 seconds
2021-11-01T17:34:04.611+01:00 END RequestId: 428afaa9-e114-43b2-8054-b71239dfb8b5
2021-11-01T17:34:04.611+01:00 REPORT RequestId: 428afaa9-e114-43b2-8054-b71239dfb8b5 Duration: 300100.73 ms Billed Duration: 300000 ms Memory Size: 128 MB Max Memory Used: 69 MB Init Duration: 364.56 ms
2021-11-01T17:35:45.040+01:00 START RequestId: 32583674-0f79-4c74-8971-07f594f8bb65 Version: $LATEST
2021-11-01T17:35:45.274+01:00 [INFO] 2021-11-01T16:35:45.274Z 32583674-0f79-4c74-8971-07f594f8bb65 Found credentials in environment variables.
2021-11-01T17:40:45.147+01:00 2021-11-01T16:40:45.147Z 32583674-0f79-4c74-8971-07f594f8bb65 Task timed out after 300.10 seconds
2021-11-01T17:40:45.147+01:00 END RequestId: 32583674-0f79-4c74-8971-07f594f8bb65
2021-11-01T17:40:45.147+01:00 REPORT RequestId: 32583674-0f79-4c74-8971-07f594f8bb65 Duration: 300101.37 ms Billed Duration: 300000 ms Memory Size: 128 MB Max Memory Used: 31 MB
Triggering rotation again gives the following error:
An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.
CDK CLI Version
1.130.0
Framework Version
1.130.0
Node.js Version
v14.17.6
OS
macOS 12.0.1
Language
Typescript
Language Version
4.4.4
Other information
No response