fix(deps-resolver): dedupePeerDependents collapses cyclic peer snapshots#1
Draft
astegmaier wants to merge 2 commits into
Draft
fix(deps-resolver): dedupePeerDependents collapses cyclic peer snapshots#1astegmaier wants to merge 2 commits into
dedupePeerDependents collapses cyclic peer snapshots#1astegmaier wants to merge 2 commits into
Conversation
…dents The pairwise strict-superset pass in deduplicateAll silently bailed on N-cycles of peer-dependency mutual references (e.g., webpack <-> terser- webpack-plugin with optional esbuild), leaving multiple snapshots of the same package in the lockfile when they only differed in propagated peer suffixes. Add a cycle-breaking phase that optimistically maps every duplicate to its richest sibling, then revalidates each tentative mapping under the *other* tentative mappings via a fixed-point loop, so an N-cycle is decided atomically rather than pairwise. Also fix a winner-deletion bug in deduplicateDepPaths that prematurely dropped the surviving winner from the unresolved set even when its group still had other surviving members it could merge with, and add chain composition at the recursion boundary so callers' single-lookup map application always returns the terminal winner. Refs: pnpm#11834 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Relocate the pnpm#11834 dedupePeerDependents cycle fix into the new pnpm11/ layout. The deduplicateAll cycle-breaking phase, the deduplicateDepPaths winner-keep change, and both tests now live under pnpm11/installing/... alongside main's determinism tie-break in depCountSorter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(deps-resolver):
dedupePeerDependentscollapses cyclic peer snapshotsFixes #11834 —
dedupePeerDependentsfailed to collapse two snapshots of the same package when the package was part of a peer-dependency cycle and one of the cycle members activated an optional transitive peer in only some workspace projects.The canonical real-world case is
webpack↔terser-webpack-pluginwith optionalesbuild: some projects in a workspace useesbuildand some don't, so pnpm produces two snapshots of each cycle member differing only in the propagated(esbuild@…)peer suffix. Both snapshots are functionally equivalent and could safely point at the richer one — butdedupePeerDependentsfailed to merge them because of how the strict pairwise check works.Tl;dr — what the fix does
Two surgical changes in
installing/deps-resolver/src/resolvePeers.ts:deduplicateAll. When the pairwise strict-superset pass stalls (produces an empty map but the duplicate groups aren't empty), we optimistically tentatively map every duplicate to its richest sibling, then drop any tentative mapping that doesn't validate under the other tentative mappings (fixed-point loop). This breaks N-cycles by deciding all cycle members atomically rather than pairwise.deduplicateDepPaths): the winner of a pairwise merge used to be removed from the "unresolved" set even though it could still be merged against other surviving winners of the same group on a later recursion. With multi-cycle workspaces (in a large monorepo at Microsoft I tested against) this hid surviving duplicates from the cycle-breaking phase. Fix: keep the winner alive in the unresolved set; only drop it when its group really has size ≤ 1.There's also a follow-up bookkeeping fix: the final returned map is chain-compressed across recursion levels so that a single lookup (
allDepPathsMap[depPath]) always returns the terminal winner.Total surface area: ~70 lines added, ~3 lines changed.
The bug, in detail
Setup
installing/deps-resolver/src/resolvePeers.tsis the final phase ofresolvePeers. After peers have been resolved, the resolver runs a deduplication pass over snapshots that share the samepkgIdWithPatchHash(i.e. same package + same patch). For each such group,deduplicateDepPathswalks pairs and asksisCompatibleAndHasMoreDeps(A, B):If yes, B is mapped to A (the strict superset). Children pointing at B are rewritten to point at A. Recurse. Repeat until either everything is merged or no progress is being made.
The reproduction (playground-pnpm-circular-peers-bug)
Two workspace projects:
packages/adepends onwebpackandterser-webpack-plugin.packages/bdepends onwebpack,terser-webpack-plugin, andesbuild.webpackdeclaresterser-webpack-pluginas a regular dep that's also an optional peer (looping back).terser-webpack-plugindeclareswebpackas a required peer andesbuildas an optional peer.Without the fix, the lockfile contains:
Four snapshots where there should be two.
Why the existing dedup pass can't handle it
The two
webpacksnapshots differ only in theirterser-webpack-pluginchild:webpack@5.107.0's child set ={ T_plain }webpack@5.107.0(esbuild)'s child set ={ T_with_e, esbuild }The strict subset check: is
{ T_plain } ⊆ { T_with_e, esbuild }? No —T_plainisn't in the right-hand set; the depPaths are literally different.Symmetric problem for the two
terser-webpack-pluginsnapshots:T_plain's childwebpack: W_plainisn't inT_with_e's child set{ W_with_e, esbuild }.Neither pair satisfies the strict check before children are rewritten. The recursion never produces a non-empty map. The pass terminates with both groups still duplicated.
How general is this?
Any peer cycle (2-cycle, 3-cycle, longer) involving optional peers exhibits this. The 2-cycle is just the simplest instance. A large monorepo at Microsoft hits the same pattern but more elaborately:
webpack↔terser-webpack-plugincycle, fourwebpacksnapshots distinguished by combinations of@swc/core,esbuild, andwebpack-clipeers.The class of broken cases: same-pkgId snapshots whose only structural difference is a child pointing into another same-pkgId duplicate group whose merge is itself blocked by this group's merge. The check is intrinsically symmetric: neither pair can resolve until the other does.
The fix, line by line
All changes are in
installing/deps-resolver/src/resolvePeers.ts.Change 1:
deduplicateDepPaths— don't drop winners prematurelyfor (const depPaths of duplicates) { const unresolvedDepPaths = new Set(depPaths.values()) let currentDepPaths = [...depPaths].sort(depCountSorter) while (currentDepPaths.length) { const depPath1 = currentDepPaths.pop()! const nextDepPaths = [] while (currentDepPaths.length) { const depPath2 = currentDepPaths.pop()! if (isCompatibleAndHasMoreDeps(depGraph, depPath1, depPath2)) { depPathsMap[depPath2] = depPath1 - unresolvedDepPaths.delete(depPath1) + // Keep depPath1 (the winner) in unresolvedDepPaths so that, on a + // subsequent recursion of deduplicateAll, it can still be compared + // against other surviving members of the same dedup group. Without + // this, peer cycles like webpack ↔ terser-webpack-plugin leave two + // winners (W3 and W4) that never get re-considered together, and + // the dedup pass terminates prematurely. See #11834. unresolvedDepPaths.delete(depPath2) } else { nextDepPaths.push(depPath2) } } nextDepPaths.push(...currentDepPaths) currentDepPaths = nextDepPaths.sort(depCountSorter) } - if (unresolvedDepPaths.size) { + if (unresolvedDepPaths.size > 1) { remainingDuplicates.push(unresolvedDepPaths) } }Why this matters. In a large monorepo at Microsoft I tested against, there are four webpack snapshots:
webpack@5.105.3(@swc/core)webpack@5.105.3(esbuild)webpack@5.105.3(esbuild)(webpack-cli)← richestwebpack@5.105.3(webpack-cli)W3 is a strict superset of W2; W4 is a strict superset of W1. Without the fix, the pairwise pass merges
W2→W3andW1→W4, then removes bothW3andW4fromunresolvedDepPaths(the old code dropped the winners, treating the group as "resolved"). The group is dropped fromremainingDuplicates. The cycle phase never sees thatW3andW4are themselves a duplicate that could be merged together (W4 → W3).With the fix, after the pairwise pass
unresolvedDepPaths = { W3, W4 }. The group survives into the next recursion. The standard pairwise pass on{W3, W4}fails the strict check (theirwebpack-clichildren differ until that group is also resolved). That's the cue for the cycle phase to fire.The
if (unresolvedDepPaths.size > 1)guard ensures we don't push singleton groups (no pairs to compare).This is its own bug, independent of the cycle phase. Even with a perfect cycle-breaking phase, the cycle phase wouldn't have been called with the right inputs in the Microsoft monorepo because the standard pass thought it was done.
Change 2: cycle phase in
deduplicateAllWhat this does. Only when the standard pass makes zero progress AND remaining groups exist do we invoke the cycle phase. This minimizes risk: every existing test path is exercised through the unchanged standard code first. The cycle phase only sees inputs the standard pass already rejected.
After the cycle phase, we remove the mapped (losing) members from each remaining group and keep only groups with ≥ 2 survivors for the recursive call.
Change 3:
deduplicatePeerCycles— optimistic candidatesThe algorithm.
(loser → winner), runisCompatibleUnderMap(next section) — a relaxed compat check that substitutes child depPaths through the current tentative map before testing subset-containment.Why it's correct. Each surviving mapping is, by construction, a real strict-superset under the actual map that will be applied. If a mapping was wrong, validation eventually catches it (no longer compatible under the reduced map), and dropping that mapping potentially invalidates others, which the fixed-point loop catches in subsequent iterations.
Why it's general. Works for arbitrary N-cycles. Doesn't depend on the cycle members forming a specific shape. Falls back gracefully when a cycle truly can't be reduced — some mappings get dropped but the others still merge. Returns an empty map in pathological cases, in which case nothing changes (status quo).
Why it's safe. It only ever runs on groups the standard pass rejected, and only emits mappings that survive validation against the substituted-children compat check. The worst case is "no extra dedup occurs" — the lockfile remains as it was, which is the pre-fix behavior anyway.
Change 4:
isCompatibleUnderMap— strict check with substitutionThe change from the existing
isCompatibleAndHasMoreDeps. Identical structure (node-dep count check, child subset check, resolved-peer subset check) — but every child depPath is looked up throughmapbefore comparison. Ifchild.children[alias] = XandXis tentatively mapped toY, we treat the child as if it pointed atY.Why this is the right relaxation. It's precisely the substitution that will happen to the graph when
mapis committed. If the substituted-child check passes, the post-rewrite graph really will have the loser's children ⊆ the winner's children. No looser than necessary.Why this isn't dangerous for genuinely different children. If a loser has a child whose
pkgIdWithPatchHashis different from anything in the winner's children, that child won't appear inmap(it's not in any duplicate group with a winner's child), so the lookup returns the original depPath unchanged, and the subset check fails as before. The relaxation only fires when the children are themselves part of a duplicate group — exactly the case we want to handle.Change 5: chain compression at the recursion boundary
Why this is needed. Take the Microsoft monorepo in concrete terms:
{ W2 → W3, W1 → W4, … }.depGraphare rewritten: anything pointing atW2now points atW3, anything atW1now points atW4. Recurse.{W3, W4}(kept alive by fix fix(deps-resolver):dedupePeerDependentscollapses cyclic peer snapshots #1). Standard pass fails (their children'swebpack-clipeers differ until that group is resolved). Cycle phase produces{ W4 → W3, WC_simple → WC_rich, … }.{ W4 → W3, WC_simple → WC_rich, … }.Now we splat iter1's map and iter2's map together:
{ W4: W3, WC_simple: WC_rich, W2: W3, W1: W4, … }.Caller looks up
allDepPathsMap[W1]and getsW4. Wrong —W4itself was merged away in iter2. The terminal winner ofW1isW3, notW4.The fix: before returning, walk
effectiveMap(the outer iter's map) and rewrite each value throughinnerMap(the next iter's map). After composition,effectiveMap[W1] = W3. The single-lookup callers (importer direct-deps rewrite at lines 192–196 ofresolvePeers.ts) get the right answer.The children-rewrite loop right above this doesn't need chain compression because it's iterative: each level applies its own map. But the externally-applied map is single-hop, so the map itself must be chain-free.
Change 6: tiny one-line export
Required so the new unit test can call it directly without bundling all of
resolvePeers.Tests
Unit tests —
installing/deps-resolver/test/deduplicatePeerCycles.test.ts(new file)Three tests, ~150 lines total:
deduplicateAll collapses two mutually-peer-dependent snapshots even when one has an optional peer activated— constructs the minimal cycle (W_plain ↔ T_plain and W_with_e ↔ T_with_e with optional esbuild peer) and asserts the map collapses both groups to their richer members.deduplicateAll cycle phase does not collapse same-pkgId snapshots whose children genuinely differ outside the cycle— constructs a case where the optional peer is itself a different version (esbuild@0.20 vs esbuild@0.27) and asserts the cycle phase correctly rejects the merge (post-hoc validation drops the tentative mappings).deduplicateAll standard pass still wins when one duplicate is a strict superset— regression check that the cycle phase doesn't fire when the standard pass can already make progress.Run them:
E2E test —
installing/deps-installer/test/install/cyclicPeerDeduplication.ts(new file)Exercises the whole resolver pipeline through
mutateModules, using@pnpm/testing.mock-agentto intercept registry HTTP calls. Three fake packages —@pnpm.e2e/cycle-peer-a,…-b,…-c— mirror the webpack/terser/esbuild trio. Two workspace projects, one activates the optional peer, one doesn't. Asserts that exactly one snapshot of A and one snapshot of B exist afterdedupePeerDependentsruns, and they both contain the(c@1.0.0)suffix (i.e. the richer member wins).Running and debugging the e2e test
The e2e tests live in the
@pnpm/installing.deps-installerpackage. They use a Rust-builtpnpm-registrybinary for registry simulation (already built in CI).If you've never built the registry binary locally:
Debugging tips:
console.log(fs.readFileSync(path.resolve('pnpm-lock.yaml'), 'utf8'))right before theexpectcalls. The lockfile is generated in a temp dir underos.tmpdir()— you can find it withprocess.cwd()from inside the test, or setlockfileDirto a known location.console.errorinsidededuplicateAllto dump the map and the remaining duplicates. The test runs in-process so logs flow straight to the terminal.getMockAgent()returns theMockAgentinstance — setMockAgent.disableNetConnect()and addagent.assertNoPendingInterceptors()to ensure no unexpected requests.deduplicateAlland re-run; the assertions should fail with 2 snapshots of A and 2 of B — confirming the test really exercises the bug.Fixture handoff to
pnpm/registry-mockThe e2e test in this PR uses
@pnpm/testing.mock-agentand stubs the registry responses inline — it does not depend on real fixtures being published. That makes it self-contained.However, several other pnpm e2e tests use real fixture packages published from
pnpm/registry-mock. For consistency with that pattern (and to allow future tests that want to exercise the cycle through the registry-mock stack rather than the undici stack), I've prepared three minimal fixturepackage.jsonfiles in this PR's session-state folder:To stage them for the registry-mock repo, run something like:
(Adjust the exact
packages/…directory layout to match whatregistry-mockuses — at the time of writing it's a flat tree ofpackage.jsonfiles underfixtures/orpackages/.)This handoff is optional for landing the current PR.
Verification — what I ran and what I saw
Repository tests
Playground reproduction
~/projects/playground-pnpm-circular-peers-bug(the original minimal repro from pnpm#11834):Real-world test — a large monorepo at Microsoft
A large internal monorepo (~5400 transitive packages) that hit this bug hard and had to apply a
packageExtensionsworkaround to force-promoteesbuildeverywhere.The one snapshot per package is the maximally-decorated one (with all peer suffixes applied), which is the safe choice.
Linking the local build to your consumer project (for manual testing)
The pnpm repo ships an officially-supported developer workflow for exactly this. Use it.
Option A (strongly recommended): the
pddev binaryThe repo's
CONTRIBUTING.mddescribes a tiny helper package atpnpm/dev/that:@pnpm/*imports straight at the sourcesrc/index.tsfiles, so it picks up your live TypeScript edits with nopnpm --filter pnpm run compilestep in between.--pm-on-fail=ignore, which bypasses anypackageManagerfield mismatch automatically — noCOREPACK_*env vars needed.One-time setup:
Then in any consumer project:
A few important properties:
pdreports itself as version1100.0.10(the marker inpnpm/dev/package.json), which is how you confirm you're not accidentally hitting a globalpnpmbinary.package.json— thepackageManager: "pnpm@…"field is silently honored-or-ignored bypd's--pm-on-fail=ignoreflag.pdinvocation re-bundles via esbuild (typically ~5s for cold, ~1s for warm cache). No manual rebuild step.Verifying you're running the local build:
pd --version # → 1100.0.10For a stronger guarantee, add a temporary
console.error('[DEBUG] local pd is running')near the top ofinstalling/deps-resolver/src/resolvePeers.tsand watch for it inpd install --lockfile-onlyoutput. Becausepdreads from source, no rebuild is needed for the marker to appear.Option B (alternative): run the bundled CLI directly with
nodeUseful if you don't want a global
pdinstall or you want to test the bundled code path (which is what real users run):Unlike
pd, this does require a rebuild (pnpm --filter pnpm run compile) after each source change, and requires you to bypass thepackageManagercheck yourself.Option C (alternative): symlink onto PATH
Same as B but lets you type
pnpm-localinstead of the longnode …/pnpm.mjsinvocation:Option D (corepack-aware, via tarball + local HTTP server)
If you want corepack to manage the local build the same way it manages a real release — i.e., keep the
packageManagerfield in the consumer project'spackage.jsonand have corepack itself fetch & cache the local binary — you can use a custom URL with corepack'sCOREPACK_ENABLE_UNSAFE_CUSTOM_URLS=1escape hatch.Important constraint. corepack documents two custom URL forms:
.jsURL → "interpreted as a CommonJS module". This does not work for pnpm, because the bundledpnpm/dist/pnpm.mjsis ESM ("type": "module"inpnpm/package.json)..tgzURL → "interpreted as a package, and thebinfield ofpackage.jsonwill be used to determine which file to use in the archive". This is the path that works.Also: corepack uses Node's
fetch()to download the URL, which does not acceptfile://URLs (tested — errors withTypeError: fetch failedfrom undici). So you have to serve the tarball over HTTP (any local server works).Verified working recipe:
Iteration friction. corepack caches the resolved URL under
~/.cache/node/corepack/v1/pnpm/<urlencoded URL>/. If you change the source and re-pack, the cached binary will be reused unless you either:package.json. Corepack will re-fetch when the hash doesn't match the cached entry.rm -rf ~/.cache/node/corepack/v1/pnpm/http*.For a tight edit-build-test loop, this is more friction than Option A's "just run
pd install". Use Option D only when you want to faithfully test the corepack-managed code path itself (e.g., to confirm the binary works end-to-end through corepack's verification gates).Option E (Verdaccio/local registry): publish the patched build
Heavier weight. Only worth it if you're testing in many separate consumer projects. Use
pnpm-deploy-local-registryorverdaccio— beyond scope here.Verifying it's actually your build
Print the version:
Or add a temporary
console.error('[DEBUG] my local build is running')near the top ofinstalling/deps-resolver/src/resolvePeers.ts, rebuild, and look for it ininstalloutput.Restoring the consumer project after testing
cd /path/to/monorepo git checkout pnpm-lock.yaml pnpm-workspace.yaml package.json(Or undo whatever backups you made. If you used
/path/to/monorepo/{pnpm-workspace,package,pnpm-lock}.yaml.bak, the.bakfiles were already restored to their original names as part of this PR's verification.)Tradeoff analysis — why this approach over the alternatives
This was the option I went with after considering five alternatives. Briefly:
isCompatibleAndHasMoreDeps; post-hoc validation would have to revert children rewrites for dropped mappings, complex)Option A's key property: it only ever runs on inputs the standard pass already rejected, and only emits mappings that pass a stricter post-hoc check. The worst case is "no extra dedup occurs" — which is exactly the pre-fix behavior. There's no path by which it can make the output worse.
pacquet parity
pacquet/AGENTS.mdrequires user-visible behavior changes in pnpm to land in pacquet too. Not applicable here: pacquet'sdedupe_peer_dependentsconfig field is plumbed through (pacquet/crates/config/src/lib.rs:596,pacquet/crates/config/src/workspace_yaml.rs:156) but the actual dedup pass inpacquet/crates/resolving-deps-resolver/src/resolve_peers.rshasn't been ported fromresolvePeers.tsyet — the file ends with peer resolution proper, nodeduplicateAll/deduplicateDepPathsequivalent. When pacquet ports the dedup pass, the port should include the cycle-breaking phase from this PR.I verified this by grepping
pacquet/fordeduplicate_all/dedupe_peer_dependents: the only matches in code paths beyond config are test files that pass the flag through.Changeset
Patch bump rationale: this is a bug fix, not a new feature or behavior change. No user-facing API changes. Lockfile shapes change in the direction the user already asked for (fewer duplicates when
dedupePeerDependents: true); no migration needed.Files changed
installing/deps-resolver/src/resolvePeers.tsdeduplicatePeerCycles,isCompatibleUnderMap; new cycle phase branch indeduplicateAll; fixed winner-deletion indeduplicateDepPaths; chain-compression after recursion; exporteddeduplicateAllfor testsinstalling/deps-resolver/test/deduplicatePeerCycles.test.tsinstalling/deps-installer/test/install/cyclicPeerDeduplication.ts.changeset/dedupe-peer-cycles.md@pnpm/installing.deps-resolverandpnpmWritten by an agent (Copilot CLI, claude-opus-4.7-1m-internal).