Skip to content

fix: don't overwrite node_modules inside inject dependencies#4299

Merged
zkochan merged 1 commit into
mainfrom
fix-inj
Feb 4, 2022
Merged

fix: don't overwrite node_modules inside inject dependencies#4299
zkochan merged 1 commit into
mainfrom
fix-inj

Conversation

@zkochan

@zkochan zkochan commented Feb 4, 2022

Copy link
Copy Markdown
Member

No description provided.

@zkochan zkochan merged commit 7ae349c into main Feb 4, 2022
@zkochan zkochan deleted the fix-inj branch February 4, 2022 22:52
Eyalm321 added a commit to Eyalm321/pnpm that referenced this pull request May 15, 2026
…t scanDir

When `runLifecycleHooksConcurrently` re-imports a workspace package into
its injected target dirs after running prepare/postinstall, it has to
preserve the existing `node_modules/` inside each target — it was set up
during the initial install (bin symlinks, transitive deps) and the
post-script re-import only ships the workspace source.

The previous approach was a manual `scanDir` over the existing
`<targetDir>/node_modules`, adding each absolute path into the re-import's
`filesMap`. That worked in 2022 (pnpm#4299) when the importer always used a
staging-temp + rename, but pnpm#11088 made the fast path (write directly into
the final dir) the default. The fast path's `makeEmptyDirSync(targetDir)`
wipes the very files the scanDir entries point at, so
`importIndexedDir → tryImportIndexedDir` crashes on the first copy with:

    ERR_PNPM_ENOENT on <targetDir>/node_modules/.bin/<tool>

Reproduces on any workspace package that has a `prepare`/`postinstall`
script and any dep with a bin entry, when injected via
`injectWorkspacePackages: true` or `dependenciesMeta.<dep>.injected`.

The fix passes `keepModulesDir: true` when an injected target already
has a `node_modules/`, which routes the import through the staging path.
`importIndexedDir`'s `moveOrMergeModulesDirs` then atomically moves the
existing `node_modules` into the staging dir — the same mechanism the
hoisted node-linker has been using since pnpm#4299. The manual scanDir is
removed; `moveOrMergeModulesDirs` covers the same ground without the
race against `makeEmptyDirSync`.

Regression test added in `injectLocalPackages.ts`: an injected workspace
package with a `prepublishOnly` script + `@pnpm.e2e/hello-world-js-bin`
dep. Before the fix this throws `ERR_PNPM_ENOENT` on
`.bin/hello-world-js-bin`; after the fix both the script output and the
bin symlink survive into the injected copy.
Eyalm321 added a commit to Eyalm321/pnpm that referenced this pull request May 15, 2026
…ipts

When `runLifecycleHooksConcurrently` re-imports a workspace package into
its injected targets after running prepare/postinstall, it has to
preserve the existing `node_modules/` inside each target — set up
during the initial install (bin symlinks, transitive deps) — while
overlaying the script's output (typically a freshly-built `dist/`).

The previous approach was a manual `scanDir` over `<targetDir>/node_modules`
that fed absolute paths into the re-import's `filesMap`. That worked in
2022 (pnpm#4299) when `importIndexedDir` always staged-and-renamed, but
pnpm#11088 made the fast path (write directly into the final dir) the
default. The fast path's `makeEmptyDirSync(targetDir)` wipes the very
files the scanned entries point at, so `tryImportIndexedDir`'s first
copy throws:

    ERR_PNPM_ENOENT on <targetDir>/node_modules/.bin/<tool>

Reproduces on any workspace package that has a `prepare`/`postinstall`
script + a dep with a bin entry, when injected via
`injectWorkspacePackages: true` or `dependenciesMeta.<dep>.injected`.

Fix: skip the `importPackage` round-trip entirely. After lifecycle
scripts complete, iterate the source's `filesMap` (from `fetchFromDir`,
which already excludes `node_modules/`) and copy each entry directly
into each `targetDir`. The target's existing `node_modules/` is never
touched — its bin links + transitive deps stay intact — and there's no
tmp-dir staging swap to trip over `.bin/<tool>` symlinks.

Regression test in `injectLocalPackages.ts`: an injected workspace
package with a `prepublishOnly` script + `@pnpm.e2e/hello-world-js-bin`
dep. Before the fix this throws `ERR_PNPM_ENOENT` on
`.bin/hello-world-js-bin`; after the fix both the script's output and
the bin symlink survive into the injected copy.
zkochan pushed a commit to Eyalm321/pnpm that referenced this pull request May 17, 2026
…ipts

When `runLifecycleHooksConcurrently` re-imports a workspace package into
its injected targets after running prepare/postinstall, it has to
preserve the existing `node_modules/` inside each target — set up
during the initial install (bin symlinks, transitive deps) — while
overlaying the script's output (typically a freshly-built `dist/`).

The previous approach was a manual `scanDir` over `<targetDir>/node_modules`
that fed absolute paths into the re-import's `filesMap`. That worked in
2022 (pnpm#4299) when `importIndexedDir` always staged-and-renamed, but
pnpm#11088 made the fast path (write directly into the final dir) the
default. The fast path's `makeEmptyDirSync(targetDir)` wipes the very
files the scanned entries point at, so `tryImportIndexedDir`'s first
copy throws:

    ERR_PNPM_ENOENT on <targetDir>/node_modules/.bin/<tool>

Reproduces on any workspace package that has a `prepare`/`postinstall`
script + a dep with a bin entry, when injected via
`injectWorkspacePackages: true` or `dependenciesMeta.<dep>.injected`.

Fix: skip the `importPackage` round-trip entirely. After lifecycle
scripts complete, iterate the source's `filesMap` (from `fetchFromDir`,
which already excludes `node_modules/`) and copy each entry directly
into each `targetDir`. The target's existing `node_modules/` is never
touched — its bin links + transitive deps stay intact — and there's no
tmp-dir staging swap to trip over `.bin/<tool>` symlinks.

Regression test in `injectLocalPackages.ts`: an injected workspace
package with a `prepublishOnly` script + `@pnpm.e2e/hello-world-js-bin`
dep. Before the fix this throws `ERR_PNPM_ENOENT` on
`.bin/hello-world-js-bin`; after the fix both the script's output and
the bin symlink survive into the injected copy.
Eyalm321 pushed a commit to Eyalm321/pnpm that referenced this pull request May 17, 2026
Per the PR review (zkochan); pairs with the turbo-side root-cause fix
vercel/turborepo#12819.

Bug 1 (peer-variant resolution): normalize once at the read layer in
convertToLockfileObject — when a pruned lockfile dropped the base
packages entry, synthesize the directory resolution from the file:
depPath (reconstruction of exactly what pnpm's writer emits, not a
guess). This makes inheritOrSynthesizeResolution + its 5 reader call
sites inert idempotent guards (resolution is always populated before
they run); the active normalization now lives solely in the converter.
Mechanical removal of the now-dead helper is a trivial follow-up.

Bug 2 (lifecycle re-import .bin ENOENT): drop the pnpm#4299 scanDir-into-
filesMap workaround and pass keepModulesDir: true so importIndexedDir
skips the pnpm#11088 makeEmptyDir fast path and preserves the target's
existing node_modules via its staging/move path. Stays on
storeController.importPackage so source files keep hardlinks — replaces
the copy-loop mirror.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions Bot pushed a commit to Eyalm321/pnpm that referenced this pull request May 18, 2026
…ipts

When `runLifecycleHooksConcurrently` re-imports a workspace package into
its injected targets after running prepare/postinstall, it has to
preserve the existing `node_modules/` inside each target — set up
during the initial install (bin symlinks, transitive deps) — while
overlaying the script's output (typically a freshly-built `dist/`).

The previous approach was a manual `scanDir` over `<targetDir>/node_modules`
that fed absolute paths into the re-import's `filesMap`. That worked in
2022 (pnpm#4299) when `importIndexedDir` always staged-and-renamed, but
pnpm#11088 made the fast path (write directly into the final dir) the
default. The fast path's `makeEmptyDirSync(targetDir)` wipes the very
files the scanned entries point at, so `tryImportIndexedDir`'s first
copy throws:

    ERR_PNPM_ENOENT on <targetDir>/node_modules/.bin/<tool>

Reproduces on any workspace package that has a `prepare`/`postinstall`
script + a dep with a bin entry, when injected via
`injectWorkspacePackages: true` or `dependenciesMeta.<dep>.injected`.

Fix: skip the `importPackage` round-trip entirely. After lifecycle
scripts complete, iterate the source's `filesMap` (from `fetchFromDir`,
which already excludes `node_modules/`) and copy each entry directly
into each `targetDir`. The target's existing `node_modules/` is never
touched — its bin links + transitive deps stay intact — and there's no
tmp-dir staging swap to trip over `.bin/<tool>` symlinks.

Regression test in `injectLocalPackages.ts`: an injected workspace
package with a `prepublishOnly` script + `@pnpm.e2e/hello-world-js-bin`
dep. Before the fix this throws `ERR_PNPM_ENOENT` on
`.bin/hello-world-js-bin`; after the fix both the script's output and
the bin symlink survive into the injected copy.
github-actions Bot pushed a commit to Eyalm321/pnpm that referenced this pull request May 18, 2026
Per the PR review (zkochan); pairs with the turbo-side root-cause fix
vercel/turborepo#12819.

Bug 1 (peer-variant resolution): normalize once at the read layer in
convertToLockfileObject — when a pruned lockfile dropped the base
packages entry, synthesize the directory resolution from the file:
depPath (reconstruction of exactly what pnpm's writer emits, not a
guess). This makes inheritOrSynthesizeResolution + its 5 reader call
sites inert idempotent guards (resolution is always populated before
they run); the active normalization now lives solely in the converter.
Mechanical removal of the now-dead helper is a trivial follow-up.

Bug 2 (lifecycle re-import .bin ENOENT): drop the pnpm#4299 scanDir-into-
filesMap workaround and pass keepModulesDir: true so importIndexedDir
skips the pnpm#11088 makeEmptyDir fast path and preserves the target's
existing node_modules via its staging/move path. Stays on
storeController.importPackage so source files keep hardlinks — replaces
the copy-loop mirror.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zkochan pushed a commit to Eyalm321/pnpm that referenced this pull request May 19, 2026
…ipts

When `runLifecycleHooksConcurrently` re-imports a workspace package into
its injected targets after running prepare/postinstall, it has to
preserve the existing `node_modules/` inside each target — set up
during the initial install (bin symlinks, transitive deps) — while
overlaying the script's output (typically a freshly-built `dist/`).

The previous approach was a manual `scanDir` over `<targetDir>/node_modules`
that fed absolute paths into the re-import's `filesMap`. That worked in
2022 (pnpm#4299) when `importIndexedDir` always staged-and-renamed, but
pnpm#11088 made the fast path (write directly into the final dir) the
default. The fast path's `makeEmptyDirSync(targetDir)` wipes the very
files the scanned entries point at, so `tryImportIndexedDir`'s first
copy throws:

    ERR_PNPM_ENOENT on <targetDir>/node_modules/.bin/<tool>

Reproduces on any workspace package that has a `prepare`/`postinstall`
script + a dep with a bin entry, when injected via
`injectWorkspacePackages: true` or `dependenciesMeta.<dep>.injected`.

Fix: skip the `importPackage` round-trip entirely. After lifecycle
scripts complete, iterate the source's `filesMap` (from `fetchFromDir`,
which already excludes `node_modules/`) and copy each entry directly
into each `targetDir`. The target's existing `node_modules/` is never
touched — its bin links + transitive deps stay intact — and there's no
tmp-dir staging swap to trip over `.bin/<tool>` symlinks.

Regression test in `injectLocalPackages.ts`: an injected workspace
package with a `prepublishOnly` script + `@pnpm.e2e/hello-world-js-bin`
dep. Before the fix this throws `ERR_PNPM_ENOENT` on
`.bin/hello-world-js-bin`; after the fix both the script's output and
the bin symlink survive into the injected copy.
zkochan pushed a commit to Eyalm321/pnpm that referenced this pull request May 19, 2026
Per the PR review (zkochan); pairs with the turbo-side root-cause fix
vercel/turborepo#12819.

Bug 1 (peer-variant resolution): normalize once at the read layer in
convertToLockfileObject — when a pruned lockfile dropped the base
packages entry, synthesize the directory resolution from the file:
depPath (reconstruction of exactly what pnpm's writer emits, not a
guess). This makes inheritOrSynthesizeResolution + its 5 reader call
sites inert idempotent guards (resolution is always populated before
they run); the active normalization now lives solely in the converter.
Mechanical removal of the now-dead helper is a trivial follow-up.

Bug 2 (lifecycle re-import .bin ENOENT): drop the pnpm#4299 scanDir-into-
filesMap workaround and pass keepModulesDir: true so importIndexedDir
skips the pnpm#11088 makeEmptyDir fast path and preserves the target's
existing node_modules via its staging/move path. Stays on
storeController.importPackage so source files keep hardlinks — replaces
the copy-loop mirror.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zkochan added a commit that referenced this pull request May 19, 2026
…th + lifecycle re-import (#11662)

Fixes two **independent** crashes hitting `pnpm install --frozen-lockfile` on workspaces with `injectWorkspacePackages: true` (or `dependenciesMeta.*.injected`), surfaced via `turbo prune --docker` pipelines.

## Bug 1 — peer-variant snapshot missing `resolution` (lean, defense-in-depth)

A peer-variant injected workspace snapshot (`@scope/pkg@file:packages/pkg(peerA@1)(peerB@2)`) inherits its `resolution` from the base `packages:` entry (`@scope/pkg@file:packages/pkg`). When a tool prunes the lockfile and drops that base entry, readers that deref `pkgSnapshot.resolution` crash with the cryptic:

```
Cannot use 'in' operator to search for 'directory' in undefined
```

**The root cause is upstream of pnpm**: the pruner (e.g. `turbo prune`) emits an internally inconsistent lockfile. Fixed at the source in **vercel/turborepo#12825** (retain the base entry for peer-variant injected deps; minimal repro in **vercel/turborepo#12824**) — empirically verified to produce a correct pruned lockfile for a real multi-service workspace.

**pnpm side (this PR): one lean normalization at the read layer** — in `convertToLockfileObject`, where base→variant inheritance already happens via `Object.assign`. When the base entry is absent, reconstruct the directory resolution from the `file:` depPath. This is *reconstruction, not guessing*: for a workspace `file:` dep the directory **is** the depPath suffix — exactly what pnpm's own writer emits. It is **defense-in-depth, not load-bearing**: with a well-formed lockfile (turbo#12825 or any correct input) the branch never fires. Because the normalization sits at the single shared read layer, it also covers the sibling `Cannot use 'in' operator … 'integrity' in undefined` on the `pnpm deploy` path (same `resolution === undefined` root, different deref site).

Per review feedback: the earlier per-reader `inheritOrSynthesizeResolution` helper across 5 call sites is **removed**; normalization lives in exactly one place (`convertToLockfileObject`), and the readers are back to `main`.

## Bug 2 — lifecycle re-import wipes `.bin/<tool>` (pure pnpm; the substantive fix)

`runLifecycleHooksConcurrently` re-imports an injected workspace package into its targets after `prepare`/`postinstall`. The 2022 `scanDir`-into-`filesMap` workaround (#4299) fed target-internal paths to `importPackage`; once #11088 made `importIndexedDir`'s `makeEmptyDir` fast path the default, that path wipes the target's `node_modules` before copying, so the re-import dies with `ERR_PNPM_ENOENT` on `node_modules/.bin/<tool>`.

Fix: drop the `scanDir` workaround and pass `keepModulesDir: true` so `importIndexedDir` skips the destructive fast path and preserves the target's existing `node_modules` (bin symlinks + transitive deps) via its staging/move path. Stays on `storeController.importPackage`, so source files keep their **hardlinks** (no copy-loop regression). Net reduction vs `main`: the `scanDir` helper and the `node:fs` / `FilesMap` imports are removed.

## Tests

- The `deps-restorer` regression fixture `peer-variant-missing-resolution` **omits the base `packages:` entry**, so it encodes the actual pruned shape and reproduces the crash on `main`: reverting the `convertToLockfileObject` change yields `resolution: undefined` for the peer-variant (→ the `lockfileToDepGraph` crash); with this PR it is reconstructed as `{ type: 'directory', directory: … }`.
- A `lockfile.fs` unit test pins the heuristic boundary: a directory resolution is synthesized for a pruned `file:` peer-variant but **never** for a `file:` tarball.
- A `deps-installer` regression test covers the Bug 2 re-import (injected dep with a `prepare` script + a bin-having dependency).

## Validation

End-to-end on a real `injectWorkspacePackages` monorepo (`turbo prune --docker` → `pnpm install --frozen-lockfile`), on services that crash on **both** bugs with stock pnpm:

- pnpm with both fixes: the crashing services build.
- **vercel/turborepo#12825 + pnpm with only Bug 2** (Bug 1 fully reverted): the crashing services still **build** → confirms Bug 1 here is genuine defense-in-depth and turbo#12825 owns the root cause.
- Bug 2 reproduces on stock pnpm regardless of turbo (it is purely pnpm's importer fast-path).

Pairs with **vercel/turborepo#12825** (Bug 1 root cause; minimal repro **vercel/turborepo#12824**). Tracks #11663.

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
Co-authored-by: Eyalm321 <eyal@sunsationsusa.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: UApply Developer <developer@uapply.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant