Skip to content

feat(core): lock graph creation when running in another process#29408

Merged
FrozenPandaz merged 1 commit intomasterfrom
file-lock-graph
Jan 28, 2025
Merged

feat(core): lock graph creation when running in another process#29408
FrozenPandaz merged 1 commit intomasterfrom
file-lock-graph

Conversation

@AgentEnder
Copy link
Copy Markdown
Member

@AgentEnder AgentEnder commented Dec 18, 2024

Current Behavior

Running Nx in multiple processes at the same time with the daemon disabled can cripple a system due to excess memory usage when creating the graph. This is due to plugin workers being started per-parent process when there is no daemon. This change enables a file lock to prevent the simultaneous processing, and read from the cache when the first run completes.

Currently, running nx show projects 30 times in parallel looks something like this:

30 processes exited within 37535ms

Expected Behavior

30 processes exited within 6435ms

Test Script

//@ts-check

const { spawn } = require('child_process');

let alive = new Set();

let start = Date.now();
let iterations = 30;

for (let i = 0; i < iterations; i++) {
  const cp = spawn('npx nx show projects', [], {
    shell: true,
    env: {
      ...process.env,
      NX_DAEMON: 'false',
      NX_VERBOSE_LOGGING: 'true',
    },
  });
  alive.add(i);
  //   cp.stdout.on('data', (data) => {
  //     console.log(`stdout [${i}]: ${data}`);
  //   });
  cp.stderr.on('data', (data) => {
    console.error(`stderr [${i}]: ${data}`);
  });
  cp.on('exit', (code) => {
    console.log(`child process ${i} exited with code ${code}`);
    alive.delete(i);
  });
}

const i = setInterval(() => {
  if (alive.size > 0) {
  } else {
    clearInterval(i);
    console.log(
      `${iterations} processes exited within ${Date.now() - start}ms`
    );
  }
}, 1);

@AgentEnder AgentEnder requested a review from a team as a code owner December 18, 2024 22:33
@AgentEnder AgentEnder requested a review from xiongemi December 18, 2024 22:33
@vercel
Copy link
Copy Markdown

vercel Bot commented Dec 18, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
nx-dev ✅ Ready (Inspect) Visit Preview Jan 24, 2025 4:54pm

@nx-cloud
Copy link
Copy Markdown
Contributor

nx-cloud Bot commented Dec 18, 2024 •
edited
Loading

View your CI Pipeline Execution ↗ for commit 73bb882.

Command Status Duration Result
nx affected --targets=lint,test,build,e2e,e2e-c... ✅ Succeeded 5m 37s View ↗
nx run-many -t check-imports check-commit check... ✅ Succeeded 25s View ↗
nx-cloud record -- nx-cloud conformance:check ✅ Succeeded 1s View ↗
nx-cloud record -- nx format:check --base=1c8b3... ✅ Succeeded 2s View ↗
nx-cloud record -- nx sync:check ✅ Succeeded <1s View ↗
nx documentation --no-dte ✅ Succeeded <1s View ↗

☁️ Nx Cloud last updated this comment at 2025-01-24 18:13:11 UTC

@AgentEnder AgentEnder force-pushed the file-lock-graph branch 2 times, most recently from d1f6a60 to 8bb1c07 Compare December 19, 2024 16:14
FrozenPandaz
FrozenPandaz requested changes Dec 19, 2024
View reviewed changes
Comment thread packages/nx/src/project-graph/project-graph.ts Outdated
Comment thread packages/nx/src/utils/file-lock.ts Outdated
Comment thread packages/nx/src/utils/file-lock.ts Outdated
@AgentEnder AgentEnder requested a review from a team as a code owner December 19, 2024 20:16
@AgentEnder AgentEnder requested a review from Cammisuli December 19, 2024 20:16
FrozenPandaz
FrozenPandaz requested changes Dec 19, 2024
View reviewed changes
Comment thread packages/nx/src/native/utils/file_lock.rs
Comment thread packages/nx/src/native/utils/file_lock.rs Outdated
Comment thread packages/nx/src/project-graph/project-graph.ts Outdated
Comment thread packages/nx/src/project-graph/project-graph.ts Outdated
@AgentEnder AgentEnder requested a review from vsavkin as a code owner December 19, 2024 23:20
@AgentEnder AgentEnder force-pushed the file-lock-graph branch 3 times, most recently from 0204e02 to beff91a Compare January 15, 2025 17:56
@AgentEnder AgentEnder requested review from a team as code owners January 15, 2025 18:06
@AgentEnder AgentEnder force-pushed the file-lock-graph branch 2 times, most recently from 16db699 to 7c67700 Compare January 16, 2025 19:06
FrozenPandaz
FrozenPandaz requested changes Jan 17, 2025
View reviewed changes
Comment thread packages/nx/src/native/utils/file_lock.rs
Comment thread packages/nx/src/native/tests/file-lock.spec.ts Outdated
Comment thread packages/nx/src/project-graph/project-graph.ts Outdated
@AgentEnder AgentEnder requested a review from a team as a code owner January 23, 2025 15:23
@AgentEnder AgentEnder requested a review from jaysoo January 23, 2025 15:23
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jan 24, 2025

Failed to publish a PR release of this pull request, triggered by @AgentEnder.
See the failed workflow run at: https://github.com/nrwl/nx/actions/runs/12956983810

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jan 24, 2025

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx create-nx-workspace@0.0.0-pr-29408-73bb882 my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-29408-73bb882
Release details 📑
Published version 0.0.0-pr-29408-73bb882
Triggered by @AgentEnder
Branch file-lock-graph
Commit 73bb882
Workflow run 12956983810

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@FrozenPandaz FrozenPandaz merged commit 5721ea3 into master Jan 28, 2025
@FrozenPandaz FrozenPandaz deleted the file-lock-graph branch January 28, 2025 14:46
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 3, 2025

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Feb 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@FrozenPandaz FrozenPandaz FrozenPandaz approved these changes

@xiongemi xiongemi Awaiting requested review from xiongemi xiongemi was automatically assigned from nrwl/nx-core-reviewers

@Cammisuli Cammisuli Awaiting requested review from Cammisuli Cammisuli was automatically assigned from nrwl/nx-native-reviewers

@vsavkin vsavkin Awaiting requested review from vsavkin

@jaysoo jaysoo Awaiting requested review from jaysoo jaysoo was automatically assigned from nrwl/nx-js-reviewers

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AgentEnder @FrozenPandaz