feat: smarter install dedupe behavior#11502
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds an opt-in config flag ChangesSmart Auto-Dedupe
Sequence DiagramsequenceDiagram
participant User
participant Install as Install<br/>Handler
participant Resolver as Dependency<br/>Resolver
participant SmartDedupe as Smart<br/>Auto-Dedupe
participant Graph as Dependency<br/>Graph
User->>Install: run install with smartAutoDedupe=true
Install->>Resolver: resolveDependencies(opts)
Resolver->>Graph: build dependency graph
Graph-->>Resolver: graph built
alt smartAutoDedupe enabled
Resolver->>SmartDedupe: applySmartAutoDedupe(graph)
SmartDedupe->>SmartDedupe: group nodes by name, peer-hash, patch-hash
SmartDedupe->>SmartDedupe: find higher in-graph versions satisfying spec
SmartDedupe->>Graph: rewrite edges and prune orphaned snapshots
Graph-->>SmartDedupe: updated graph
SmartDedupe-->>Resolver: void (in-place)
end
Resolver-->>Install: resolved packages
Install-->>User: install complete
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint install failed. For unrecoverable errors, disable the tool in CodeRabbit configuration. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “smart auto dedupe” pass to restore safer automatic subdependency deduplication during pnpm install (intended to recover behavior lost after the deterministic resolution change in #11110), by backtracking over the resolved dependency graph and rewriting eligible parent→child edges to higher in-graph versions that still satisfy the original requesting spec.
Changes:
- Introduces a new
applySmartAutoDedupe()graph-rewrite pass in@pnpm/installing.deps-resolver, gated behindsmartAutoDedupe. - Captures per-package dependency spec ranges (
depSpecs) during resolution to validate dedupe rewrites against the requesting manifest ranges. - Plumbs the new flag through config/CLI → installer → resolver, and adds both unit + integration tests plus a changeset.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| installing/deps-resolver/test/smartAutoDedupe.test.ts | Adds unit tests for the new graph rewrite logic and safety constraints (peers, patches, non-registry resolutions, etc.). |
| installing/deps-resolver/src/smartAutoDedupe.ts | Implements the smart auto dedupe pass and resolution eligibility checks. |
| installing/deps-resolver/src/resolveDependencyTree.ts | Threads smartAutoDedupe into resolver context/options. |
| installing/deps-resolver/src/resolveDependencies.ts | Adds optional depSpecs capture on resolved packages when the feature is enabled. |
| installing/deps-resolver/src/index.ts | Invokes applySmartAutoDedupe() when enabled and exposes the option on resolver entrypoint. |
| installing/deps-installer/test/install/dedupe.ts | Adds integration tests asserting lockfile pruning/retention behavior with the feature on/off. |
| installing/deps-installer/src/install/index.ts | Passes smartAutoDedupe into resolveDependencies() from installer options. |
| installing/deps-installer/src/install/extendInstallOptions.ts | Adds the new option to strict install options with a default of false. |
| installing/commands/src/installDeps.ts | Allows the install-deps command options to pick up smartAutoDedupe from config. |
| installing/commands/src/install.ts | Allows the install command options to pick up smartAutoDedupe from config. |
| config/reader/src/types.ts | Registers smart-auto-dedupe as a boolean config key. |
| config/reader/src/index.ts | Adds default value for smart-auto-dedupe. |
| config/reader/src/configFileKey.ts | Categorizes smart-auto-dedupe among excluded YAML config keys. |
| config/reader/src/Config.ts | Adds smartAutoDedupe?: boolean to the config shape. |
| .changeset/smart-auto-dedupe.md | Declares the new opt-in feature and publishes minor bumps across affected packages. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
372d319 to
4f4f415
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
installing/deps-installer/src/install/index.ts (1)
1286-1301:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
smartAutoDedupecan be skipped by frozen/headless install paths.Line 1300 forwards the flag, but the same file’s frozen/full-resolution gating (e.g., Line 817 and Lines 514-519) does not consider
smartAutoDedupe. With an up-to-date lockfile, resolution may be skipped entirely, so smart dedupe never runs.💡 Proposed fix
- let needsFullResolution = outdatedLockfileSettings || + let needsFullResolution = outdatedLockfileSettings || opts.fixLockfile || !upToDateLockfileMajorVersion || opts.forceFullResolution || - forceResolutionFromHook + forceResolutionFromHook || + opts.smartAutoDedupe- !opts.dedupe && + !opts.dedupe && + !opts.smartAutoDedupe &&
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.changeset/smart-auto-dedupe.md:
- Line 9: Update the release note for the smartAutoDedupe feature to explicitly
document the registry/patch guardrails: state that smartAutoDedupe
(smart-auto-dedupe = true) only rewrites edges between dedupable registry
tarball entries (not git/workspace/local tarballs) and will only rewrite when
the target snapshot has the same patch hash (in addition to matching resolved
peer set and satisfying the original spec range), so users understand
patched/git/workspace/local entries are not eligible for rewrites.
In `@installing/deps-resolver/src/smartAutoDedupe.ts`:
- Around line 120-123: The semver.validRange check on parent.depSpecs[alias]
fails for aliased specifiers like "npm:`@scope/pkg`@^1.0.0"; update the logic
around parent.depSpecs[alias] (the code using semver.validRange, spec,
findUpgrade, child.version, childDepPath) to normalize or strip alias wrappers
before calling semver.validRange (e.g., remove "npm:" and any alias syntax) so
aliased ranges are parsed correctly, or alternatively normalize depSpecs earlier
when populated in resolveDependencies.ts so parent.depSpecs contains plain
semver ranges for semver.validRange to succeed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3124ff01-b42d-4c2e-bacd-dd76f5b5a10f
📒 Files selected for processing (16)
.changeset/smart-auto-dedupe.mdconfig/reader/src/Config.tsconfig/reader/src/configFileKey.tsconfig/reader/src/index.tsconfig/reader/src/types.tscspell.jsoninstalling/commands/src/install.tsinstalling/commands/src/installDeps.tsinstalling/deps-installer/src/install/extendInstallOptions.tsinstalling/deps-installer/src/install/index.tsinstalling/deps-installer/test/install/dedupe.tsinstalling/deps-resolver/src/index.tsinstalling/deps-resolver/src/resolveDependencies.tsinstalling/deps-resolver/src/resolveDependencyTree.tsinstalling/deps-resolver/src/smartAutoDedupe.tsinstalling/deps-resolver/test/smartAutoDedupe.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- installing/deps-installer/src/install/extendInstallOptions.ts
- config/reader/src/Config.ts
- installing/deps-resolver/src/index.ts
Adds an opt-in `smartAutoDedupe` setting that runs a backtracking pass over the resolved dependency graph: for every parent → child edge, if a higher version of the same package already exists in the graph (sharing the same peer-dep-graph hash and patch hash, both registry-resolved) and that higher version satisfies the original spec range, the edge is rewritten to point at the higher version. Orphaned snapshots are then pruned from the lockfile. The pass is intentionally conservative: only registry-tarball resolutions participate, peer/patch boundaries are never crossed, non-semver specs are skipped, and importer-level direct refs are left untouched. Restores the automatic dedupe behavior removed in pnpm#11110 without reintroducing the non-determinism that motivated that fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- exclude git-hosted tarballs from dedupe (gitHosted flag, isGitHostedPkgUrl, and non-semver depPaths) so a registry edge is never rewritten to a git-hosted depPath - unwrap npm: alias specifiers before the semver range check so aliased edges can be deduped - skip the optimistic frozen fast-path when smartAutoDedupe is enabled so a plain install re-resolves and runs the pass, while leaving explicit --frozen-lockfile installs untouched - clarify the changeset activation (pnpm-workspace.yaml / --smart-auto-dedupe) and document the registry/patch/peer/git/workspace/local guardrails - rename a misleading test; add regression tests for the above 🤖 Generated with [Claude Code](https://claude.com/claude-code) (Opus 4.8) Co-Authored-By: Claude <noreply@anthropic.com>
4f4f415 to
aba8553
Compare
Code Review by Qodo
1.
|
Only merge graph nodes that share the same prod/dev/optional reachability signature. Previously a required parent→child edge could be rewritten onto a node that was reached only through optional (or dev-only) edges, leaving the now-required package marked `optional: true` (or dev-only) in the lockfile — which suppresses its fetch/install failures and makes it prunable under --prod. Since the rewrite target now always carries the exact reachability of the redirected edge, no node's flags become stale. 🤖 Generated with [Claude Code](https://claude.com/claude-code) (Opus 4.8) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/reader/src/configFileKey.ts`:
- Line 94: Add support for the `smart_auto_dedupe` configuration field to match
pnpm's implementation. Add `smart_auto_dedupe: Option<bool>` field to the
`WorkspaceSettings` struct, include it in the `clear_workspace_only_fields()`
function to ensure it is properly cleared as a workspace-only field, and add the
corresponding `smart_auto_dedupe` field with an appropriate default value to the
`Config` struct to maintain parity with pnpm's behavior for this configuration
option.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 87408f48-043b-45e8-8a92-5ec76268eaa5
📒 Files selected for processing (5)
.changeset/smart-auto-dedupe.mdconfig/reader/src/Config.tsconfig/reader/src/configFileKey.tsconfig/reader/src/index.tsconfig/reader/src/types.ts
💤 Files with no reviewable changes (1)
- config/reader/src/types.ts
✅ Files skipped from review due to trivial changes (1)
- config/reader/src/Config.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- config/reader/src/index.ts
| 'dedupe-peers', | ||
| 'dedupe-direct-deps', | ||
| 'dedupe-injected-deps', | ||
| 'smart-auto-dedupe', |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Read-only parity check: confirm pacquet mirrors the newly excluded key.
set -euo pipefail
echo "Searching pnpm side:"
rg -n --type=ts -C2 "'smart-auto-dedupe'" config/reader/src/configFileKey.ts config/reader/src/types.ts config/reader/src/index.ts
echo
echo "Searching pacquet mirror paths for the same key/filter:"
rg -n -C3 "smart-auto-dedupe|excludedPnpmKeys|clear_workspace_only_fields|isConfigFileKey" pacquet
echo
echo "Inspecting the parity function block (if present):"
fd workspace_yaml.rs pacquet --exec sed -n '540,660p' {}Repository: pnpm/pnpm
Length of output: 16360
🏁 Script executed:
# Check if smart_auto_dedupe field exists in WorkspaceSettings and whether it's being cleared
echo "=== Searching for smart_auto_dedupe in pacquet config ==="
rg -n "smart_auto_dedupe|smart-auto-dedupe" pacquet/crates/config/src/
echo
echo "=== Full clear_workspace_only_fields function ==="
sed -n '595,650p' pacquet/crates/config/src/workspace_yaml.rs
echo
echo "=== WorkspaceSettings struct definition to see all fields ==="
rg -B5 -A80 "^pub struct WorkspaceSettings" pacquet/crates/config/src/workspace_yaml.rs | head -120Repository: pnpm/pnpm
Length of output: 7181
🏁 Script executed:
# Check if smart_auto_dedupe field is defined anywhere in pacquet WorkspaceSettings
echo "=== Full WorkspaceSettings struct to find smart_auto_dedupe ==="
rg -A200 "^pub struct WorkspaceSettings" pacquet/crates/config/src/workspace_yaml.rs | grep -i "smart\|dedupe"
echo
echo "=== Check if Config struct has smart_auto_dedupe ==="
rg -n "smart_auto_dedupe|smart-auto-dedupe" pacquet/crates/config/src/lib.rs
echo
echo "=== Check all dedupe fields in Config struct ==="
rg -B2 -A2 "dedupe.*:" pacquet/crates/config/src/lib.rs | head -50Repository: pnpm/pnpm
Length of output: 2195
Pacquet is missing the smart-auto-dedupe field entirely—add it to maintain parity.
The new config key 'smart-auto-dedupe' added to pnpm's excludedPnpmKeys (forbidding it in global config.yaml) is not defined in pacquet's WorkspaceSettings or Config structs at all. Users setting smart-auto-dedupe in pnpm-workspace.yaml will have it silently ignored in pacquet, and if the field is later added without the clear_workspace_only_fields() filter, pacquet would allow it in global config while pnpm forbids it. Add smart_auto_dedupe: Option<bool> to WorkspaceSettings, include it in clear_workspace_only_fields(), and add the corresponding field and default to Config to match pnpm's behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@config/reader/src/configFileKey.ts` at line 94, Add support for the
`smart_auto_dedupe` configuration field to match pnpm's implementation. Add
`smart_auto_dedupe: Option<bool>` field to the `WorkspaceSettings` struct,
include it in the `clear_workspace_only_fields()` function to ensure it is
properly cleared as a workspace-only field, and add the corresponding
`smart_auto_dedupe` field with an appropriate default value to the `Config`
struct to maintain parity with pnpm's behavior for this configuration option.
Source: Coding guidelines
|
Code review by qodo was updated up to the latest commit 0c82d83 |
…#12472) When a package is reused from the lockfile, its child edges are taken verbatim and bypass the preferred-versions walk, so a transitive dependency can stay pinned to an older version even after a direct dependency resolved to a higher version that satisfies the same range — leaving the lockfile non-convergent (an incremental install keeps a duplicate that a fresh install would not). The resolver now refreshes such a stale pin to the higher direct-dependency version during resolution, via `preferredVersion` (singular), which overrides the EXISTING_VERSION_SELECTOR_WEIGHT stability bias. The older version is never resolved or fetched, and the incremental result converges to what a fresh install produces. The pick is anchored to direct dependencies (which resolve first), so it restores the automatic dedupe removed in #11110 without reintroducing its non-determinism, and unlike the post-pass in #11502 it does not over-fetch. pacquet is ported in the same change. Its full-subtree lockfile reuse is coarser than pnpm's per-edge reuse, so it records per importer which direct deps changed and their resolved versions, declines full-subtree reuse for a parent that depends on a changed direct dep, and forces the higher version in the child walk. Range satisfaction uses plain semver (not prerelease-inclusive), matching pnpm's semver.satisfies(.., true).
we've been seeing this non-deterministic update behavior for a long while now - running
pnpm dedupeand usingauto-install-peers=falsehave helped but we're still seeing this impact core transitive dependencies that are AFAICT unrelated to the newly installed packages. This happens for both peer dependency resolutions and version pinning so a dependency can either switch from v1.0.1 -> v1.0.2 randomly or v1.0.1(react@18) -> v1.0.1 with no peer dep.from a comment on #11110 (comment), I would have expected that to be the default behavior.
disclaimer: the vast majority of this was claude generated - the overall idea is I think solid but I'm assuming there are gaps in implementation, please let me know 😁
Summary by CodeRabbit
Summary by CodeRabbit
smart-auto-dedupe(disabled by default). When enabled, pnpm performs a conservative post-resolution deduplication pass to rewrite dependency edges to higher compatible in-graph versions and prunes orphaned lockfile snapshots.