Skip to content

(cli): hotswap results in many call to ListStackResources of cloudformation API (with ThrottlingException) #19021

@JPLemelin

Description

@JPLemelin

What is the problem?

We are using the hot-swap with lambda code change (cdk deploy --hotswap my-stack-with-lambda-functions --exclusively) .
Currently the stack have 68 lambdas, who are all using the same code asset, but with different entry point.

The problem we are facing is, the process take almost the exact same time of cloudformation deploy, i.e around 5-6mins 🥲

Reproduction Steps

Have a stack with many lambda, in my case 68 lambda

What did you expect to happen?

Only one call to ListStackResources of cloudformation API (or more if paged)

What actually happened?

Foreach lambdas, a ListStackResources of cloudformation API are done and some hit the ThrottlingException

CDK CLI Version

2.12.0

Framework Version

No response

Node.js Version

14

OS

MacOs

Language

Typescript

Language Version

No response

Other information

Hi!

We are using the hot-swap with lambda code change (cdk deploy --hotswap my-stack-with-lambda-functions --exclusively) .
Currently the stack have 68 lambdas, who are all using the same code asset, but with different entry point.

The problem we are facing is, the process take almost the exact same time of cloudformation deploy, i.e around 5-6mins 🥲

After some investigation we found the cdk do some (many many many 🙃) api call to cloudforamtion on ListStackResources, who take around 5mins.

On the 876 call to ListStackResources, 604 end with ThrottlingException: Rate exceeded

On first few second there are 68 call (same number of lambda) to ListStackResources who don’t have nextToken parameter, I think it’s the first call to ListStackResources, other call have a paging parameters (nextToken

)

So most of the hot-swap execution time is spent doing ListStackResources

Our cloudformation stack have 359 ressources

When we use the cli, aws cloudformation list-stack-resources --stack-name my-stack-with-lambda-functions , we did not hit any paging

I also add some screen shot from cloudtrail of CDK call during the hot-swap

image
image
image

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.p2package/toolsRelated to AWS CDK Tools or CLI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions