@aws-cdk/aws-s3: CloudFormation stack deployment fails intermittently when adding a bucket policy and EventBridge notifications #27600
Description
Describe the bug
CloudFormation stack deployment fails intermittently when adding a bucket policy (PutBucketPolicy) and EventBridge notifications (PutBucketNotification) to a S3 bucket due to race condition.
Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (OperationAborted) when calling the PutBucketNotificationConfiguration operation: A conflicting conditional operation is currently in progress against this resource. Please try again.. See the details in CloudWatch Log Stream: 2023/10/17/[$LATEST]9f69597966xxxxa8449646270045 (RequestId: 2xx08c-74ad-4317-8a30-83xxxf2dc9)
Below is my CDK code of using S3 Bucket construct with eventBridgeEnabled: true and adding the policy after creating the bucket.
export class S3Bucket extends s3.Bucket {
constructor(scope: Construct, id: string, props: S3BucketProperties) {
super(scope, id, {
...props,
removalPolicy: cdk.RemovalPolicy.DESTROY,
encryption: s3.BucketEncryption.KMS,
encryptionKey: props.encryptionKey,
eventBridgeEnabled: true,
});
}
}
---------------------------------------------
this.s3Bucket = new S3Bucket(this, 'bucket', {
encryptionKey,
});
this.s3Bucket.addToResourcePolicy(
createS3BucketSSLRequestsOnlyPolicyStatement(
this.s3Bucket.bucketArn
)
);
Similar issue #16811
Expected Behavior
S3 Bucket policy (PutBucketPolicy) and bucket notifications (PutBucketNotification/PutBucketNotificationConfiguration) happen without failing.
Current Behavior
Stack deployment fails intermittently
Reproduction Steps
Below is my CDK code of using S3 Bucket construct with eventBridgeEnabled: true and adding the policy after creating the bucket.
export class S3Bucket extends s3.Bucket {
constructor(scope: Construct, id: string, props: S3BucketProperties) {
super(scope, id, {
...props,
removalPolicy: cdk.RemovalPolicy.DESTROY,
encryption: s3.BucketEncryption.KMS,
encryptionKey: props.encryptionKey,
eventBridgeEnabled: true,
});
}
}
---------------------------------------------
this.s3Bucket = new S3Bucket(this, 'bucket', {
encryptionKey,
});
this.s3Bucket.addToResourcePolicy(
createS3BucketSSLRequestsOnlyPolicyStatement(
this.s3Bucket.bucketArn
)
);
Possible Solution
(AWS Support Recommended) Implement retry with incremental back off into the custom resource code. This includes retrying operations with an exponential back off time to cater for any issues.
Additional Information/Context
No response
CDK CLI Version
2.99.1
Framework Version
No response
Node.js Version
18
OS
MacOS
Language
TypeScript
Language Version
No response
Other information
No response