Skip to content

astegmaier/playground-pnpm-circular-peers-bug

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dedupePeerDependents doesn't deduplicate when there's a cyclic relationship between packages

When a package's regular dependency forms a peer cycle with it (the dep declares a required peer back on its parent) and declares an optional peer that some consumers activate but others don't, pnpm creates two parent snapshots and dedupePeerDependents fails to merge them.

Repro

Workspace with two projects: one has webpack, the other has webpack and esbuild. Neither package declares the other as a peer. Both want a single shared webpack instance after dedupe.

pnpm-workspace.yaml

packages:
  - 'packages/*'
dedupePeerDependents: true     # default in pnpm 10+; pinned for clarity

packages/a/package.json

{
  "name": "a",
  "dependencies": {
    "esbuild": "^0.27.0",
    "webpack": "^5.107.0"
  }
}

packages/b/package.json

{
  "name": "b",
  "dependencies": {
    "webpack": "^5.107.0"
  }
}

webpack's package.json

{
  "name": "webpack",
  "dependencies": { 
    "terser-webpack-plugin": "^5.3.10" // terser-webpack-plugin points back to webpack as a required peer
    // ...
  }
}

terser-webpack-plugin's package.json

{
  "name": "terser-webpack-plugin",
  "peerDependencies": { 
    "webpack": "^5.1.0" // <-- webpack points back at terser-webpack-plugin as a regular dependency
  },
  "peerDependenciesMeta": {
    "esbuild":   { "optional": true }
    // ...
  }
}

That's a peer cycle: webpack → (regular dep) → terser-webpack-plugin → (peer) → webpack. terser-webpack-plugin also declares esbuild as an optional peer, which is the divergent thing between consumer a (has esbuild in scope) and consumer b (doesn't).

Expected pnpm-lock.yaml (only one webpack snapshot)

snapshots:
  webpack@5.107.0(esbuild@0.27.7):
    dependencies:
      terser-webpack-plugin: 5.6.0(esbuild@0.27.7)(webpack@5.107.0)
      ...

This seems to be the behavior the dedupePeerDependents docs promise for projects that share a dep and differ only on an optional peer: pick the richer snapshot, point both projects at it.

Actual pnpm-lock.yaml (two webpack snapshots)

snapshots:
  webpack@5.107.0:
    dependencies:
      terser-webpack-plugin: 5.6.0(webpack@5.107.0)
      ...
    transitivePeerDependencies:
      - esbuild
      - ...
  webpack@5.107.0(esbuild@0.27.7):
    dependencies:
      terser-webpack-plugin: 5.6.0(esbuild@0.27.7)(webpack@5.107.0)
      ...
    transitivePeerDependencies:
      - esbuild
      - ...

b/node_modules/webpack symlinks to the first; a's symlinks to the second. The dedupe pass left both snapshots in place.

Why we'd expect dedupePeerDependents to handle this

From the docs:

let's say we have a workspace with two projects and both of them have webpack in their dependencies. webpack has esbuild in its optional peer dependencies, and one of the projects has esbuild in its dependencies. In this case, pnpm will link two instances of webpack to the node_modules/.pnpm directory: one with esbuild and another one without it… you may now use the dedupePeerDependents setting to deduplicate webpack when it has no conflicting peer dependencies… both projects will use the same webpack instance, which is the one that has esbuild resolved.

The doc states the case precisely:

  • Two workspace projects each depend on the same package (here webpack).
  • The two resulting snapshots differ only on whether an optional peer is resolved.
  • They have no conflicting peers.
  • With dedupePeerDependents: true, they should collapse to the richer one.

That's exactly the situation in this repro. The only difference from the doc's example is where the optional peer is declared (the doc's statement that 'esbuild' is optional peer of 'webpack' is inaccurate)

Related issues

  • #9427Issue with webpack in monorepo - TypeError: The 'compilation' argument must be an instance of Compilation (open). Same runtime symptom and same "two webpack snapshots in .pnpm/" cause, surfaced by ModuleFederationPlugin.getCompilationHooks, just as here, but no repro was provided.
  • #6154Dependency resolution not deterministic (closed, but reopened in a comment from April 2023 with the same 'compilation' argument must be an instance of Compilation error against webpack@5.78.0_esbuild@0.16.17 vs webpack@5.79.0_esbuild@0.16.17 — exactly this repro's (esbuild@…) peer-suffix pattern).
  • #6171dedupe-peer-dependents sometimes doesn't (closed as fixed in 7.29.0-2). Uses essentially the same package combination (webpack + terser-webpack-plugin) as this repro; the 2023 fix didn't cover the optional-peer-through-cycle variant our repro exercises.
  • #7600dedupe-peer-dependents not working as expected (open). zkochan's comment acknowledges the algorithmic gap and suggests changing parentPkgs in resolvePeers.ts; that change never landed.

Why dedupePeerDependents actually fails: a peer cycle hides the divergence one level down

Although webpack appears to be the package with the divergent optional peer (the (esbuild@…) suffix is on webpack's snapshot key), it isn't. webpack's own package.json declares no esbuild peer:

{
  "peerDependencies": null,
  "peerDependenciesMeta": {}
}

The (esbuild@…) in the snapshot key is propagated up via transitivePeerDependencies from one of webpack's regular dependencies: terser-webpack-plugin, whose package.json declares:

{
  "peerDependencies": { "webpack": "^5.1.0" },
  "peerDependenciesMeta": {
    "esbuild":   { "optional": true },
    "@swc/core": { "optional": true },
    "uglify-js": { "optional": true }
  }
}

Two things relevant to dedupe:

  1. terser-webpack-plugin peers on webpack, and webpack pulls in terser-webpack-plugin as a regular dep. That's a peer cycle: webpack → (regular dep) → terser-webpack-plugin → (peer) → webpack.
  2. esbuild is an optional peer of terser-webpack-plugin — when the importer has esbuild in scope, the optional peer activates and the snapshot key picks up (esbuild@…).

With a providing esbuild and b not, terser splits into two snapshots, and webpack (which depends on terser as a regular child) is cloned into two snapshots to match — propagating the esbuild choice up via transitivePeerDependencies.

Where deduplicateAll gives up

installing/deps-resolver/src/resolvePeers.tsdeduplicateAlldeduplicateDepPathsisCompatibleAndHasMoreDeps:

const node1DepPathsSet = new Set(Object.values(node1.children!))
const node2DepPaths = Object.values(node2.children!)
if (!node2DepPaths.every((depPath) => node1DepPathsSet.has(depPath))) return false

For two webpack snapshots to merge, every child of the loser must appear in the winner's child set. They don't, because their terser-webpack-plugin children point at different terser snapshots:

webpack snapshot terser-webpack-plugin child snapshot
webpack@5.107.0(esbuild@…) terser-webpack-plugin@5.6.0(esbuild@…)(webpack@5.107.0)
webpack@5.107.0 terser-webpack-plugin@5.6.0(webpack@5.107.0)

Could dedupe collapse the terser snapshots first and retry? The same check fails symmetrically there because each terser snapshot's webpack peer-child points back at the matching webpack snapshot — the cycle:

terser-webpack-plugin snapshot webpack peer-child
terser-webpack-plugin@5.6.0(esbuild@…)(webpack@5.107.0) webpack@5.107.0(esbuild@…)
terser-webpack-plugin@5.6.0(webpack@5.107.0) webpack@5.107.0

A single pass can't break the cycle. deduplicateAll only recurses when the first pass produces some mapping; here it produces zero and the algorithm stops with no progress.

What pnpm's doc example misses

The doc puts the optional esbuild peer directly on webpack. In that shape there's no intermediate package and no cycle: two webpack snapshots, identical children apart from the optional peer, dedupe trivially.

When the optional peer is declared on a regular dep that also peers back into its parent, the snapshot divergence isn't local to one node — it's threaded through a cycle that the single-pass child-subset check can't unwind. The doc's behavior contract still applies (only optional peer differs, no conflicting peers), so it reads as a doc bug or an implementation bug depending on perspective.

Workaround

Promote the optional peer to a regular dep via packageExtensions:

packageExtensions:
  'terser-webpack-plugin@^5.0.0':
    dependencies:
      esbuild: '*'

With esbuild always present as a real child of terser-webpack-plugin, the optional peer no longer adds a snapshot-key suffix, the propagation through transitivePeerDependencies stops, the cycle no longer matters, and dedupePeerDependents collapses webpack to a single snapshot.

This isn't a satisfying fix — esbuild is now a hard dependency of every project that pulls webpack in, even when nothing uses esbuild — but it works and is the only pnpm-workspace.yaml-only fix we found.

Version history: this is a regression introduced in pnpm 7.33.1

dedupePeerDependents did not always mishandle this case. When the setting first shipped in pnpm 7.29.0 (#6153) it deduplicated this exact webpack/esbuild cycle correctly — both projects shared a single webpack@…(esbuild@…) instance. The behavior regressed in 7.33.1 and has been broken in every release since, up to and including the latest tested (11.9.0).

Testing each version by deleting the lockfile, running a fresh pnpm install, and counting distinct webpack peer contexts in the resulting pnpm-lock.yaml:

pnpm version lockfile webpack contexts result
7.29.0 (first release with the setting) v5.4 1 ✅ deduped
7.31.0, 7.32.0, 7.33.0 v5.4 1 ✅ deduped
7.33.1 (first broken) v5.4 2 ❌ duplicated
7.33.2, 7.33.3, 7.33.7 (latest 7.x) v5.4 2 ❌ duplicated
8.15.9 (latest 8.x) v6.0 2 ❌ duplicated
9.15.9 (latest 9.x) v9.0 2 ❌ duplicated
10.34.4 (latest 10.x) v9.0 2 ❌ duplicated
11.9.0 (latest) v9.0 2 ❌ duplicated

✅ deduped = one webpack snapshot that both projects share (the expected result); ❌ duplicated = two webpack snapshots, i.e. the bug. Bisection pins the split to the 7.33.0 → 7.33.1 boundary, and the single resolver change in that range is the root cause below. By construction, then, every release from 7.29.0 through 7.33.0 is free of the bug — they are also all long past end-of-life, so no supported pnpm avoids it.

Note: older pnpm 7.x releases (e.g. 7.29.0) throw ERR_INVALID_THIS when fetching the npm registry on Node 20+, so the early-7.x runs above were done on Node 18. The split is version-driven, not Node-driven: 7.33.7 duplicates on both Node 18 and Node 24.

Root cause: #6606

The regression came from commit b7820a0"fix(resolvePeers): fix deduplication when version missmatch" (#6606, closing #6605). It changed how isCompatibleAndHasMoreDeps (the same check quoted in Where deduplicateAll gives up) compares two candidate snapshots' children:

- const node1DepPaths = Object.keys(node1.children)    // child aliases (package names)
- const node2DepPaths = Object.keys(node2.children)
+ const node1DepPaths = Object.values(node1.children)  // child dep-paths (incl. peer suffix)
+ const node2DepPaths = Object.values(node2.children)

node.children is a { alias → depPath } map, so the one-word change from keys to values flipped the comparison from names to resolved paths:

  • Before (≤ 7.33.0): compatibility compared child names (Object.keys). webpack@5.107.0 and webpack@5.107.0(esbuild@…) have the same set of child names (terser-webpack-plugin, acorn, …), so the richer variant was judged "compatible and has more deps" and the two collapsed into one.
  • After (≥ 7.33.1): compatibility compares child dep-paths (Object.values). The two webpack variants' terser-webpack-plugin children resolve to different dep-paths (one carries the (esbuild@…) peer suffix, one doesn't), so the strict check now returns false and the variants are never merged.

The change correctly fixed #6605 (comparing by name wrongly merged children whose versions differed), but it also made the comparison reject the optional-peer-through-a-cycle case — where the differing child dep-path is exactly the divergence that dedupePeerDependents is supposed to absorb. The same Object.values(node1.children) check is still present today in installing/deps-resolver/src/resolvePeers.ts (isCompatibleAndHasMoreDeps), so the bug has survived the lockfile format moving from v5.4 → v9.0 and the file relocating from pkg-manager/resolve-dependencies/ to installing/deps-resolver/.

What are the symptoms of this failure to deduplicate?

The runtime symptom typically surfaces as a TypeError from a webpack plugin (see Related issues above):

TypeError: The 'compilation' argument must be an instance of Compilation
    at NormalModule.getCompilationHooks (.../webpack@5.107.0/.../NormalModule.js:265:10)
    at .../packages/b/webpack.config.js:11:20
    ...
    at Compiler.newCompilation (.../webpack@5.107.0_esbuild@0.27.7/.../Compiler.js:1342:30)

The two divergent webpack paths in the stack are the smoking gun: NormalModule was loaded from webpack@5.107.0, but compilation was constructed by webpack@5.107.0_esbuild@0.27.7. Two distinct Compilation class identities → instanceof Compilation inside getCompilationHooks fails → throw.

To trigger this from the repro:

cd packages/b
pnpm build

The build script in b runs bundle (the bin exposed by a), whose require('webpack') walks from a/bundle.jsa's subtree → the esbuild webpack variant. That bundle then require()s b/webpack.config.js; the plugin in the config calls require('webpack') which walks from b/webpack.config.jsb's subtree → the no-esbuild webpack variant. Two different webpacks at runtime, instanceof Compilation fails:

const { NormalModule } = require('webpack');

class TouchCompilationHooksPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap('TouchCompilationHooksPlugin', (compilation) => {
      NormalModule.getCompilationHooks(compilation);
    });
  }
}

module.exports = { entry: './src/index.js', plugins: [new TouchCompilationHooksPlugin()] };

Any webpack plugin that calls one of webpack's getCompilationHooks class methods will fail the same way when the build is driven by a different webpack instance than the one loaded by the plugin — which is precisely what the duplicate snapshots cause. The repro uses NormalModule; different plugins surface the same root cause from different call sites (see Related issues for two real-world examples — ModuleFederationPlugin and webpack-manifest-plugin).

About

A reproduction of a bug in dedupePeerDependents in pnpm

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors