Skip to content

fix(js): improve @nx/js/typescript plugin and typescript-sync generator performance#28379

Merged
JamesHenry merged 1 commit intomasterfrom
js/plugin-sync-generator-perf
Oct 15, 2024
Merged

fix(js): improve @nx/js/typescript plugin and typescript-sync generator performance#28379
JamesHenry merged 1 commit intomasterfrom
js/plugin-sync-generator-perf

Conversation

@leosvelperez
Copy link
Copy Markdown
Member

@leosvelperez leosvelperez commented Oct 9, 2024 •
edited
Loading

  • Significantly improve the perf of the @nx/js/typescript plugin (see below) by not using the calculateHashForCreateNodes function to create the cache key. That function is very generic and always hashes all project files to create the cache key. The @nx/js/typescript plugin output is solely influenced by the TS configuration files and installed packages, nothing else. So, this PR replaces the usage of calculateHashForCreateNodes with a more surgical hash of what really matters.
  • Refactor the cache handling of the @nx/js:typescript-sync sync generator. Additionally, cache the checks for file existence.

Before

The @nx/js/typescript plugin execution times out after 10 minutes (on my machine) running in the repo: https://github.com/leosvelperez/huge-ts-monorepo-perf-issue. It's a repo with 2251 libraries, each with about 500 TypeScript files.

NX_DAEMON=false NX_PERF_LOGGING=true NX_VERBOSE_LOGGING=true pnpm nx graph --no-open
Time for 'loading dotenv files' 0.28191699999996445
Time for 'workspace context init' 1.7878329999999778
Time for 'Load Nx Plugin: /Users/leosvel/code/playground/huge-ts-monorepo/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/plugins/project-json/build-nodes/project-json' 0.3739999999999952
Time for 'Load Nx Plugin: /Users/leosvel/code/playground/huge-ts-monorepo/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/plugins/package-json' 0.6995000000000005
Time for 'Load Nx Plugin: /Users/leosvel/code/playground/huge-ts-monorepo/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/plugins/js' 15.443958999999992
Time for 'Load Nx Plugin: @nx/js/typescript' 433.81937500000004
Time for 'nx/js/dependencies-and-lockfile:createNodes' 72.19974999999977
Time for 'loadNxPlugins' 571.003625
Time for 'nx/core/project-json:createNodes' 0.1109580000047572
Time for 'nx/core/package-json:createNodes' 2415.916250000002
Time for 'build-project-configs' 600072.1975
Time for 'createNodes:merge' 26.039540999918245
Time for 'native-file-deps' 0.0012499999720603228
Time for 'get-workspace-files' 1451.049417000031
Time for 'get-all-workspace-files' 360.20758299995214
Time for 'read cache' 0.1965000000782311
Time for 'build typescript dependencies' 37416.08045799995
Time for 'nx/js/dependencies-and-lockfile:createDependencies' 40340.08241699997

 NX   Failed to create nodes

An error occurred while processing files for the @nx/js/typescript plugin.
  - @nx/js/typescript timed out after 10 minutes during createNodes. As a last resort, you can set NX_PLUGIN_NO_TIMEOUTS=true to bypass this timeout.
    Error: @nx/js/typescript timed out after 10 minutes during createNodes. As a last resort, you can set NX_PLUGIN_NO_TIMEOUTS=true to bypass this timeout.
        at Timeout.<anonymous> (/Users/leosvel/code/playground/huge-ts-monorepo/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/project-graph/plugins/isolation/plugin-pool.js:219:21)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7)
...

After

The @nx/js/typescript plugin in about 15 seconds in the same repo:

NX_DAEMON=false NX_PERF_LOGGING=true NX_VERBOSE_LOGGING=true pnpm nx graph --no-open
Time for 'loading dotenv files' 0.2621250000000259
Time for 'workspace context init' 1.7320000000000277
Time for 'Load Nx Plugin: /Users/leosvel/code/playground/huge-ts-monorepo-perf/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/plugins/package-json' 0.3127500000000083
Time for 'Load Nx Plugin: /Users/leosvel/code/playground/huge-ts-monorepo-perf/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/plugins/project-json/build-nodes/project-json' 0.4220419999999905
Time for 'Load Nx Plugin: /Users/leosvel/code/playground/huge-ts-monorepo-perf/node_modules/.pnpm/nx@20.0.0-canary.20241002-1d10a19_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5._kpa22fllnv36ws73ps6dxoprny/node_modules/nx/src/plugins/js' 12.358916999999991
Time for 'Load Nx Plugin: @nx/js/typescript' 463.34191699999997
Time for 'nx/js/dependencies-and-lockfile:createNodes' 71.64483299999847
Time for 'loadNxPlugins' 600.3308750000001
Time for 'nx/core/project-json:createNodes' 0.11258400000224356
Time for 'nx/core/package-json:createNodes' 1325.8219579999932
Time for '@nx/js/typescript:createNodes' 15625.287917000001

Current Behavior

Expected Behavior

Related Issue(s)

Fixes #

@leosvelperez leosvelperez self-assigned this Oct 9, 2024
@leosvelperez leosvelperez requested review from a team as code owners October 9, 2024 11:21
@vercel
Copy link
Copy Markdown

vercel Bot commented Oct 9, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Updated (UTC)
nx-dev ⬜️ Ignored (Inspect) Visit Preview Oct 10, 2024 3:27pm

@nx-cloud
Copy link
Copy Markdown
Contributor

nx-cloud Bot commented Oct 9, 2024 •
edited
Loading

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 9d9c46c. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

✅ Successfully ran 4 targets

Sent with 💌 from NxCloud.

@leosvelperez leosvelperez force-pushed the js/plugin-sync-generator-perf branch from dbc347b to 9d9c46c Compare October 10, 2024 15:24
@JamesHenry JamesHenry merged commit 76580db into master Oct 15, 2024
@JamesHenry JamesHenry deleted the js/plugin-sync-generator-perf branch October 15, 2024 15:49
jaysoo pushed a commit that referenced this pull request Oct 15, 2024
@leosvelperez @jaysoo
fix(js): improve @nx/js/typescript plugin and typescript-sync generat…
4850905 
…or performance (#28379)
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 22, 2024

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Oct 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@JamesHenry JamesHenry JamesHenry approved these changes

@AgentEnder AgentEnder Awaiting requested review from AgentEnder AgentEnder was automatically assigned from nrwl/nx-core-reviewers

Assignees

@leosvelperez leosvelperez

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@leosvelperez @JamesHenry