fix: large amounts of metadata can break 512MB string limit by rix0rrr · Pull Request #34480 · aws/aws-cdk

rix0rrr · 2025-05-16T11:55:25Z

A large amount of stacks leads to a large body of metadata into the manifest.json file. The file can grow too large to be written to disk (>512MB), but even if it doesn't fail outright, serializing and writing and loading a large JSON file takes a lot of CPU time.

This PR splits the metadata for stacks off into a separate file which is written separately, moving it out of the hot path of working with the manifest.json file. On a tree of 256 stacks and ~40,000
constructs, this shaves off 5 seconds off of the total synthesis time.

Also in this PR:

The upstream CloudArtifact exposes a metadata getter that reads from manifest or file, whichever is available. Expose here as well for completeness.
Pull through the final thread for the stack casing change that was initiated here: fix(cli): remove legacy stack tags behavior aws-cdk-cli#1053

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

aws-cdk-automation

(This review is outdated)

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

packages/aws-cdk-lib/core/lib/private/construct-iteration.ts

aws-cdk-automation · 2025-05-26T11:21:14Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: f04ec71
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

When CDK apps grow extremely large (think 10-100 stacks, 1000-10000 constructs), all metadata together begins to exceed 512MB, the maximum string size in NodeJS. People usually deal with this by disabling metadata, but they shouldn't have to. In addition, even for manifests that don't exceed 512MB the extremely large size of the single JSON object slows down its writing and reading every time, even if the metadata doesn't need to be accessed. An effective solution is to write the metadata of an artifact to a separate file. This PR introduces the ability for that into the Cloud Assembly schema, and updates the CLI to read from both sources if available. Relates to aws/aws-cdk#34480.

When CDK apps grow extremely large (think 10-100 stacks, 1000-10000 constructs), all metadata together begins to exceed 512MB, the maximum string size in NodeJS. People usually deal with this by disabling metadata, but they shouldn't have to. In addition, even for manifests that don't exceed 512MB the extremely large size of the single JSON object slows down its writing and reading every time, even if the metadata doesn't need to be accessed. An effective solution is to write the metadata of an artifact to a separate file. This PR introduces the ability for that into the Cloud Assembly schema, and updates the CLI to read from both sources if available. Removes an unused class from the refactoring project that we found through this project because it would have to be refactored. Rather than refactoring, bye-bye! Relates to aws/aws-cdk#34480. --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit df361bb Merge: f14a874 68bde3b Author: Rico Huijbers <rix0rrr@gmail.com> Date: Fri Feb 13 13:41:58 2026 +0100 Merge branch 'huijbers/synth-perf' of github.com:aws/aws-cdk into huijbers/synth-perf commit f14a874 Merge: 5bbd63a ca4409b Author: Rico Huijbers <rix0rrr@gmail.com> Date: Fri Feb 13 13:41:23 2026 +0100 Merge remote-tracking branch 'origin/main' into huijbers/synth-perf commit 68bde3b Author: Rico Hermans <rix0rrr@gmail.com> Date: Mon May 26 13:16:07 2025 +0200 Update construct-iteration.ts Comments were outdated commit f04ec71 Author: Rico Hermans <rix0rrr@gmail.com> Date: Mon May 26 13:15:42 2025 +0200 Naming was shyte commit 5bbd63a Author: Rico Huijbers <rix0rrr@gmail.com> Date: Fri May 16 13:57:39 2025 +0200 MOve this code over from the other pR commit 046eb38 Author: Rico Huijbers <rix0rrr@gmail.com> Date: Fri May 16 13:49:51 2025 +0200 fix: large amounts of stacks slow down synthesis A large amount of stacks leads to a large body of metadata into the `manifest.json` file. The file can grow too large to be written to disk (>512MB), but even if it doesn't fail outright, serializing and writing and loading a large JSON file takes a lot of CPU time. This PR splits the metadata for stacks off into a separate file which is written separately, moving it out of the hot path of working with the `manifest.json` file. On a tree of 256 stacks and ~40,000 constructs, this shaves off 5 seconds off of the total synthesis time.

mergify · 2026-03-06T08:58:44Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2026-03-06T08:59:56Z

Merge Queue Status

Rule: default-squash

✅ Entered queue — 2026-03-06 08:58 UTC
✅ Checks started · in-place
✅ All checks passed · ready to merge 🚀
⬜ Merge

mergify · 2026-03-06T10:02:11Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions · 2026-03-06T10:02:26Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

rix0rrr requested a review from a team as a code owner May 16, 2025 11:55

aws-cdk-automation requested a review from a team May 16, 2025 11:55

github-actions bot added the p2 label May 16, 2025

mergify bot added the contribution/core This is a PR that came from AWS. label May 16, 2025

rix0rrr force-pushed the huijbers/synth-perf branch from 27b76dd to 046eb38 Compare May 16, 2025 11:56

aws-cdk-automation previously requested changes May 16, 2025

View reviewed changes

rix0rrr added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label May 16, 2025

rix0rrr changed the title ~~fix: large amounts of stacks slow down synthesis~~ fix: large amounts of stacks slows down synthesis May 16, 2025

matboros reviewed May 19, 2025

View reviewed changes

packages/aws-cdk-lib/core/lib/private/construct-iteration.ts Show resolved Hide resolved

matboros reviewed May 19, 2025

View reviewed changes

packages/aws-cdk-lib/core/lib/private/construct-iteration.ts Show resolved Hide resolved

matboros reviewed May 19, 2025

View reviewed changes

packages/aws-cdk-lib/core/lib/private/construct-iteration.ts Outdated Show resolved Hide resolved

rix0rrr self-assigned this May 26, 2025

github-actions bot mentioned this pull request Jun 1, 2025

Monthly PR metrics report #34601

Closed

rix0rrr mentioned this pull request Jun 3, 2025

feat(cli): metadata can live in a separate file aws/aws-cdk-cli#568

Merged

rix0rrr force-pushed the huijbers/synth-perf branch 2 times, most recently from 17e8374 to 6b989ed Compare February 13, 2026 12:55

rix0rrr changed the base branch from main to huijbers/generators February 13, 2026 12:56

rix0rrr force-pushed the huijbers/synth-perf branch from 6b989ed to a8aa6cb Compare February 13, 2026 12:57

rix0rrr changed the base branch from huijbers/generators to main February 13, 2026 12:57

Upgrade schema + API + fix tests

a071e56

This was referenced Feb 13, 2026

Failed CodeBuild CI - 15 PR(s) pahud/aws-cdk#17

Open

PRs with Conflicts - 3 PR(s) pahud/aws-cdk#13

Open

Merge remote-tracking branch 'origin/main' into huijbers/synth-perf

d9dce4a

rix0rrr temporarily deployed to automation March 3, 2026 13:08 — with GitHub Actions Inactive

rix0rrr changed the title ~~fix: large amounts of stacks slows down synthesis~~ fix: large amounts of metadata can 512MB string limit Mar 3, 2026

rix0rrr changed the title ~~fix: large amounts of metadata can 512MB string limit~~ fix: large amounts of metadata can break 512MB string limit Mar 3, 2026

rix0rrr temporarily deployed to automation March 3, 2026 13:48 — with GitHub Actions Inactive

rix0rrr temporarily deployed to automation March 5, 2026 12:33 — with GitHub Actions Inactive

Fix endless loop

2f1379b

rix0rrr temporarily deployed to automation March 5, 2026 12:39 — with GitHub Actions Inactive

Fix unit tests

0f7e017

rix0rrr temporarily deployed to automation March 5, 2026 13:19 — with GitHub Actions Inactive

Merge branch 'main' into huijbers/synth-perf

169383e

rix0rrr temporarily deployed to automation March 5, 2026 14:00 — with GitHub Actions Inactive

mrgrain approved these changes Mar 5, 2026

View reviewed changes

rix0rrr added the pr/do-not-merge This PR should not be merged at this time. label Mar 5, 2026

rix0rrr temporarily deployed to automation March 5, 2026 14:05 — with GitHub Actions Inactive

rix0rrr removed the pr/do-not-merge This PR should not be merged at this time. label Mar 6, 2026

rix0rrr temporarily deployed to automation March 6, 2026 08:55 — with GitHub Actions Inactive

Merge branch 'main' into huijbers/synth-perf

c7f8042

mergify bot temporarily deployed to automation March 6, 2026 09:30 Inactive

mergify bot merged commit 20c3154 into main Mar 6, 2026
16 of 17 checks passed

mergify bot deleted the huijbers/synth-perf branch March 6, 2026 10:02

github-actions bot locked as resolved and limited conversation to collaborators Mar 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: large amounts of metadata can break 512MB string limit#34480

fix: large amounts of metadata can break 512MB string limit#34480
mergify[bot] merged 7 commits intomainfrom
huijbers/synth-perf

rix0rrr commented May 16, 2025 •

edited

Loading

Uh oh!

aws-cdk-automation left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aws-cdk-automation commented May 26, 2025

Uh oh!

mergify bot commented Mar 6, 2026

Uh oh!

mergify bot commented Mar 6, 2026 •

edited

Loading

Uh oh!

mergify bot commented Mar 6, 2026

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

rix0rrr commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aws-cdk-automation left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aws-cdk-automation commented May 26, 2025

AWS CodeBuild CI Report

Uh oh!

mergify bot commented Mar 6, 2026

Uh oh!

mergify bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

mergify bot commented Mar 6, 2026

Uh oh!

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rix0rrr commented May 16, 2025 •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

mergify bot commented Mar 6, 2026 •

edited

Loading