Skip to content
This repository was archived by the owner on May 14, 2026. It is now read-only.
This repository was archived by the owner on May 14, 2026. It is now read-only.

Support nodeLinker: 'hoisted' — umbrella #438

@zkochan

Description

@zkochan

Support nodeLinker: 'hoisted' — umbrella

Tracks everything pacquet needs to do to support pnpm's hoisted node-linker mode end to end. Today the value deserializes and round-trips into .modules.yaml, but no install code branches on itpacquet install with nodeLinker: hoisted silently produces an isolated layout and a .modules.yaml that claims it is hoisted.

This is an umbrella, not a single PR. See Workstream for the independently shippable slices.

Upstream references pin pnpm/pnpm main at 94240bc046.

Not to be confused with #435. hoistPattern and publicHoistPattern are isolated-linker settings; they are not consulted at all under nodeLinker: hoisted. The two features share nothing at install time — different hoisters, different on-disk shapes, different fields populated. Track that work in #435.

Goals

  1. Correctness. pacquet install with nodeLinker: hoisted produces the same node_modules/ shape as pnpm — real directories at the project root, the hoist algorithm matching pnpm's choice of which version wins each top-level slot, conflicts placed under <parent>/node_modules/<conflict>.
  2. .modules.yaml round-trip parity. hoistedDependencies and hoistedLocations written faithfully so subsequent installs honor the same plan and rebuild knows where each depPath lives.
  3. Repeat-install semantics. Subsequent installs diff the old plan against the new and rimraf orphans, in the same order pnpm does.
  4. Workspace semantics. In a workspace, each project's hoist plan is computed jointly with the others (one shared workspace-wide tree), not per-project independently.

Non-goals

  • PnP linker. nodeLinker: 'pnp' is out of scope.
  • pnpm deploy under hoisted. Quietly disables lockfile use upstream; pacquet has no deploy command yet.
  • GVS interaction. Global virtual store + hoisted is its own animal; tracked separately under Add global virtual store support to pacquet install --frozen-lockfile #432.
  • Injected workspace packages. injectionTargetsByDepPath is populated but the re-mirror step is a separate code path.
  • pnpm install --filter subset under hoisted. Hoisting decisions must be globally consistent, so partial installs effectively don't exist under this linker. Worth noting but not implementing.

Upstream surface

Every claim cites a permalink at 94240bc046.

1. Conceptual model

Hoisted produces real directories at node_modules/<pkg>/, not symlinks. Conflicts nest under <parent>/node_modules/<conflict>. No node_modules/.pnpm/ virtual store; the only on-disk persistence besides the tree itself is .modules.yaml. Two data structures:

2. Pipeline entry points (every branch on nodeLinker === 'hoisted')

3. lockfileToHoistedDepGraph

installing/deps-restorer/src/lockfileToHoistedDepGraph.ts. Takes the wanted lockfile (+ optional current); returns {graph, prevGraph, hierarchy, directDependenciesByImporterId, hoistedLocations, injectionTargetsByDepPath, symlinkedDirectDependenciesByImporterId}. prevGraph is what linkHoistedModules diffs against to find orphans.

Internally calls hoist(lockfile, …) (real-hoist), then fetchDeps walks the HoisterResult tree:

4. Hoist algorithm (real-hoist)

installing/linking/real-hoist/src/index.ts wraps @yarnpkg/nm/hoist:

The @yarnpkg/nm hoister is not pure pnpm code. For pacquet either (a) port @yarnpkg/nm/hoist to Rust, or (b) reimplement an equivalent name-conflict-aware tree-hoister with the same observable behavior (parent wins; respect peerNames; respect hoistingLimits). Either way, the interface into the hoister is what pacquet builds — the tree shape, the workspace-as-children trick, the peer-names knob, the external-dependencies trick.

5. .modules.yaml schema (hoisted-linker fields)

installing/modules-yaml/src/index.ts:23-48. https://github.com/pnpm/pnpm/blob/94240bc046/installing/modules-yaml/src/index.ts#L23-L48

Fields that this feature reads and writes:

  • hoistedDependencies: HoistedDependencies — the plan from real-hoist.
  • hoistedLocations: Record<string, string[]>hoisted-linker only. Required by rebuild (MISSING_HOISTED_LOCATIONS) and by the skip-fetch optimization in lockfileToHoistedDepGraph.
  • nodeLinker: 'hoisted' — the persisted linker selection.
  • included: IncludedDependencies — validated against the current install; INCLUDED_DEPS_CONFLICT on mismatch.

Writer (canonical, both modes): https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/index.ts#L660-L682

6. Bin linking under hoisted

7. hoistingLimits and externalDependencies

Both knobs are inputs to real-hoist — they exist specifically for the hoisted linker.

8. Workspace semantics

One shared hoisted tree at the workspace root, plus per-project node_modules for whatever couldn't hoist up:

9. Peer-dep interaction with autoInstallPeers

autoInstallPeers: true zeroes out peerNames for the hoister, letting it move packages past parents that supply their peers. https://github.com/pnpm/pnpm/blob/94240bc046/installing/linking/real-hoist/src/index.ts#L124-L129

Tests: hoistedNodeLinker/install.ts:314 (react-dom with autoInstallPeers expects node_modules/react at top) https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/test/hoistedNodeLinker/install.ts#L314 and :329 (peer-of-self regression) https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/test/hoistedNodeLinker/install.ts#L329

10. Purge / reinstall semantics

Direct-on-disk packages can drift; hoisted does its own cleanup:

11. Modules-cache / lockfile invalidation

The lockfile-level outdated-settings detector explicitly excludes nodeLinker and hoistingLimits — switching to hoisted does NOT invalidate the lockfile. Pacquet's outdated detector must match. https://github.com/pnpm/pnpm/blob/94240bc046/lockfile/settings-checker/src/getOutdatedLockfileSetting.ts

12. Forbidden combinations / errors specific to hoisted

Pacquet's current state

# Surface Verdict Evidence
1 Config.node_linker deserialize Have crates/config/src/lib.rs:28-41 (enum), :164 (field), crates/config/src/workspace_yaml.rs:53 + :207-208 (plumb).
2 .modules.yaml round-trip of nodeLinker Have crates/package-manager/src/install.rs:242-247 (map_node_linker), crates/modules-yaml/src/lib.rs:167 (field). Round-trip tests at crates/modules-yaml/tests/index.rs:25,82,86,....
3 .modules.yaml.hoistedDependencies field Have (schema only) Field at crates/modules-yaml/src/lib.rs:155; HoistKind enum at :303. Legacy hoistedAliases migrator at :461-482. Always written empty (build_modules_manifest at crates/package-manager/src/install.rs:259-278 leaves it ..Default::default(), with the doc-comment at :254-258 calling out the gap).
4 .modules.yaml.hoistedLocations field Missing entirely No field named hoisted_locations on Modules. (grep -rn "hoisted_locations" crates/ returns nothing in the struct schema.) This field is required for rebuild under hoisted.
5 Install pipeline branches on node_linker Missing grep -rn "NodeLinker::Hoisted" crates/ outside the enum + serde glue + the map_node_linker writer returns zero hits. Hoisted installs silently produce the isolated layout.
6 lockfileToHoistedDepGraph analog Missing No equivalent of the directory-keyed DependenciesGraph builder. crates/package-manager/src/deps_graph.rs is depPath-keyed and used only for cache hashing.
7 Hoist algorithm (real-hoist) Missing No port of @yarnpkg/nm/hoist or equivalent. No hoisting_limits or external_dependencies plumbing.
8 linkHoistedModules analog Missing All linking is symlink-based (create_virtual_store.rs + symlink_direct_dependencies.rs). No real-directory import path that overwrites with force: true.
9 Bin linking from a project-root node_modules Partial link_bins.rs:30-65 (link_direct_dep_bins) writes shims to <modules_dir>/.bin from direct-dep symlinks. The hoisted path needs to do this from real directories and per-parent <dir>/node_modules/.bin per hoist location.
10 --node-linker CLI flag Missing No node-linker / node_linker references in crates/cli/src/cli_args/. Setting requires pnpm-workspace.yaml or .npmrc.
11 Workspace support Missing (foundationally) crates/package-manager/src/install.rs:95-99 documents the absence. Workspaces are a prerequisite for hoisted full-parity, but a single-importer hoisted install is shippable first.
12 bundledDependencies extraction Missing Out of scope for this issue, but referenced for completeness — BUNDLED_DEPENDENCIES_WITHOUT_HOISTED is an upstream error gating this feature on hoisted.

In one sentence: pacquet has the deserialize-and-persist surface for hoisted; everything that produces the on-disk layout is missing.

Workstream

Slices are roughly ordered. Earlier slices unblock later ones.

Slice 1 — Real-directory linking primitive

Foundational. Hoisted's import step is "copy/hardlink the package's files into <parent>/node_modules/<alias>/, possibly overwriting an existing directory". Pacquet's existing primitives all symlink. This slice ports the equivalent of storeController.importPackage with keepModulesDir: true and force: true.

Scope:

  • New function in crates/package-manager/ (or a helper in crates/store-dir/ since it consumes the CAS) that takes a CAS file index + destination path and populates the destination as real files (matching pacquet's existing import method preference: hardlink, then clonefile, then copy).
  • Overwrite semantics: any pre-existing file at the destination is replaced. Pre-existing node_modules/ subdirectory is preserved (keepModulesDir: true) so nested deps don't get clobbered between the rimraf-orphans pass and the insert pass.
  • Stand-alone unit tests in the new module — does not yet need to be wired into the install pipeline.

Slice 2 — .modules.yaml.hoistedLocations field

Required for rebuild and for the "is the package already on disk" skip-fetch optimization.

Scope:

  • Add hoisted_locations: Option<BTreeMap<DepPath, Vec<String>>> to Modules (crates/modules-yaml/src/lib.rs).
  • Round-trip tests against the upstream schema.
  • Reader returns it from a freshly-deserialized .modules.yaml; writer serializes when non-empty.
  • Update the misleading comment at crates/package-manager/src/install.rs:254-258.

Slice 3 — Hoist algorithm port (single-importer)

The load-bearing slice.

Scope:

  • Reimplement an equivalent of @yarnpkg/nm/hoist in Rust, or port the upstream wrapper around it. Public surface matches upstream's hoist(lockfile, opts): input = the lockfile + root importer's direct deps + hoistingLimits + externalDependencies + per-package peerNames. Output = a tree of (alias, depPath, children) rooted at the project.
  • Defer multi-importer / workspace input — the function takes one importer for now but should be designed to take a Map<ImporterId, …> once workspaces land.
  • Defer hoistingLimits / externalDependencies knobs — accept empty inputs for now; wire them later when a real consumer appears.
  • Unit tests against fixtures derived from upstream's installing/linking/real-hoist/test/ cases.

Slice 4 — lockfileToHoistedDepGraph analog

Walks the hoist result and produces the directory-keyed graph that the link step consumes.

Scope:

  • walk_hoist_result(hoist_root, lockfile_dir) -> { graph, hoisted_locations, direct_deps_by_importer, …}.
  • graph is keyed by absolute directory path; each node carries depPath, alias, children, the package's package.json, etc.
  • Populates hoisted_locations map for Slice 2 / 5.
  • Refuses to record a package whose installability check (see Proper optionalDependencies support — umbrella #434 Slice 1) returns false — adds to Skipped set instead.
  • Diffs against any persisted hoistedDependencies/hoistedLocations to produce a prev_graph for Slice 6's orphan diff.

Slice 5 — link_hoisted_modules (the install-time linker)

Produces the on-disk tree from Slice 4's graph using Slice 1's primitive.

Scope:

  • Walk graph and prev_graph; rimraf surplus directories first (orphan removal), then import packages second. Match tryRemoveDir's EPERM/EBUSY tolerance.
  • For each graph node, call Slice 1's import primitive with force=true, keep_modules_dir=true.
  • Per-node_modules bin link: walk children, link each one's bins into <node_modules>/.bin. Per the upstream pass, this is one call per parent directory.
  • Skip-fetch optimization: when currentHoistedLocations[depPath] includes the target directory and <dir>/package.json exists, do not refetch from CAS.

Slice 6 — Pipeline branch and Install::run wiring

Plug Slices 3–5 into the install pipeline behind a node_linker == Hoisted branch.

Scope:

  • In Install::run (install.rs) and InstallFrozenLockfile::run (install_frozen_lockfile.rs), branch on config.node_linker:
    • NodeLinker::Isolated → existing path.
    • NodeLinker::Hoisted → Slice 3 (hoist) → Slice 4 (walk) → Slice 5 (link) → existing build phase (BuildModules, which already takes a lockfile + virtual-store dir; for hoisted, rebuild needs to be retargeted to operate on hoisted_locations directories).
  • Populate Modules.hoisted_dependencies and Modules.hoisted_locations in build_modules_manifest.
  • Implement MISSING_HOISTED_LOCATIONS error for the rebuild path.

Slice 7 — Build phase under hoisted (BuildModules retargeting)

Hoisted's build phase runs over real directories from hoistedLocations, not virtual-store slots.

Scope:

  • Add a hoisted-mode entry into BuildModules (or a sibling type) that iterates hoistedLocations rather than virtual_store_dir/<slot>/node_modules/<pkg>.
  • extraBinPaths per-package is binDirsInAllParentDirs(pkgRoot, lockfileDir) — every ancestor node_modules/.bin up to lockfileDir, in lookup order. Port the helper.
  • Side-effects cache (Wire is_built skip + side-effects cache WRITE path (#397 item #10 follow-up to PR #422) #421) under hoisted: cache key shape unchanged; only the import target differs.

Slice 8 — --node-linker CLI flag

Tiny.

Scope:

  • Add --node-linker [isolated|hoisted|pnp] to install / add / update (probably the top-level CliArgs). Overrides Config.node_linker.

Slice 9 — Workspace-aware hoisting

Depends on workspace support landing in pacquet (separate roadmap item).

Scope:

  • Real-hoist tree includes every workspace importer as a child of the virtual root, named encodeURIComponent(importerId) with reference = workspace:<id>.
  • lockfileToHoistedDepGraph analog produces per-project hoist subtrees and a workspace-wide direct_deps_by_importer_id.
  • link:-direct deps go through symlink_direct_dependencies (already implemented) instead of the real-directory linker.
  • Workspace hoisted installs ignore --filter for the hoist step — globally consistent decisions only.
  • hoistWorkspacePackages (default true) plumbed.

Slice 10 — hoistingLimits and externalDependencies knobs

Programmatic-only upstream; pacquet may still want them exposed as pnpm-workspace.yaml settings for parity.

Scope:

  • Config.hoisting_limits: BTreeMap<String, BTreeSet<String>>.
  • Config.external_dependencies: BTreeSet<String>.
  • Plumbed into Slice 3's hoister inputs.

Slice 11 — Forbidden-combinations errors

Tail end. Mirror upstream's error codes via pacquet-diagnostics.

Scope:

  • ERR_PNPM_BUNDLED_DEPENDENCIES_WITHOUT_HOISTED, ERR_PNPM_MISSING_HOISTED_LOCATIONS, ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. Each at its appropriate validate-or-resolve site.

Tests to port

From plans/TEST_PORTING.md "Support nodeLinker=hoisted":

Primary:

  • installing/deps-installer/test/hoistedNodeLinker/install.ts:16 installing with hoisted node-linker
  • :45 installing with hoisted node-linker and no lockfile
  • :61 overwriting (is-positive@3.0.0 with is-positive@latest)
  • :83 overwriting existing files in node_modules
  • :97 preserve subdeps on update
  • :119 adding a new dependency to one of the workspace projects (depends on Slice 9)
  • :172 installing the same package with alias and no alias
  • :187 run pre/postinstall scripts. bin files should be linked in a hoisted node_modules
  • :210 running install scripts in a workspace that has no root project (depends on Slice 9)
  • :229 hoistingLimits should prevent packages to be hoisted (depends on Slice 10)
  • :247 externalDependencies should prevent package from being hoisted to the root (depends on Slice 10)
  • :264 linking bins of local projects when node-linker is set to hoisted (depends on Slice 9)
  • :314 peerDependencies should be installed when autoInstallPeers is set to true and nodeLinker is set to hoisted
  • :329 installing with hoisted node-linker a package that is a peer dependency of itself
  • installing/deps-installer/test/install/multipleImporters.ts:87 (depends on Slice 9)

Frozen/headless cross-coverage:

  • optionalDependencies.ts:594 includes hoisted frozen install (depends on Proper optionalDependencies support — umbrella #434)
  • patch.ts:24, :120, :297, :386 (depends on patching parity — already present)
  • lifecycleScripts.ts:579, :686
  • deps-restorer/test/index.ts:859, :873

Cross-references


Written by an agent (Claude Code, claude-opus-4-7).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions