Follow-up to #11857 / #11851. The remaining clean-install gap to pnpm CLI is architectural, not a kernel-contention problem. Spinning this out as its own tracking issue so #11857 can stay scoped to the original (now-debunked) hypothesis.

TL;DR

install_with_fresh_lockfile → install_subtree is a recursive per-package async tree walk that doesn't batch the link phase. install_frozen_lockfile → CreateVirtualStore is a phased warm/cold-batch architecture with a single rayon par_iter over snapshots. The two paths solve the same problem on disk but the recursive shape is structurally slower. Routing the fresh-lockfile path through CreateVirtualStore after resolution should close the gap.

Evidence

4-scenario sweep on the alotta-files fixture (3 343 packages, verdaccio mock), 10-core M-series Mac, 5-run hyperfine with 2 warmups:

The full-resolution-warm row pins the architectural cost: with a warm store and no lockfile, the resolve phase (~4–5 s) plus the recursive install_subtree link phase (~17 s) totals 22 s. With a lockfile, the same on-disk work goes through CreateVirtualStore and lands in 7.5 s.

Where the cost is

install_with_fresh_lockfile (the no-lockfile entry, crates/package-manager/src/install_with_fresh_lockfile.rs) does:

1. resolve_importer — walks the manifest via the resolver chain, produces the peer-resolved graph (~4-5 s)
2. prefetch_cas_paths — best-effort warm-cache batch lookup against index.db (~0 s on cold; populates for the install pass on warm)
3. build_fresh_lockfile — converts the resolved graph into the v9 snapshots: / packages: shape (~3 ms)
4. VirtualStoreLayout::new — precomputes per-snapshot slot directories
5. install_subtree — recursive per-package walk that awaits download + import + symlink before recursing into children

Step 5 is the structural problem. Each install_subtree call awaits install_package_from_registry for one package, which itself runs import_indexed_dir synchronously (blocking the tokio worker on a rayon par_iter for one destination directory). The try_join_all across siblings only buys ~10-way parallelism (= tokio worker count) and rayon work-stealing across simultaneous par_iters on different destination dirs is worse than one phased par_iter over all of them.

install_frozen_lockfile (the lockfile entry, crates/package-manager/src/install_frozen_lockfile.rs) does:

1. Lockfile parsed → packages, snapshots, importers, current_snapshots, current_packages
2. prefetch_cas_paths + integrity verify via rayon par_iter (~ms when warm)
3. compute_skipped_snapshots for platform-mismatched optionals + previously-skipped entries
4. VirtualStoreLayout::new
5. CreateVirtualStore::run — single phased pass that does:
   • warm batch: rayon par_iter over snapshots whose CAS paths the prefetch already verified, calling CreateVirtualDirBySnapshot::run (import + symlink layout via rayon::join)
   • cold batch: try_join_all of downloads, falling into the same shape once the tarball is in CAS
6. SymlinkDirectDependencies::run — creates the node_modules/<alias> → slot symlinks for every importer's direct deps in one batch
7. LinkVirtualStoreBins::run — creates <slot>/node_modules/.bin/* per slot using the prefetched manifests

The CreateVirtualStore warm batch is a single rayon par_iter over all snapshots, with one work-stealing scope. The fresh-lockfile path's recursive install_subtree is N nested par_iters each scoped to one package's files. Same total CPU, very different rayon scheduling behavior.

What the refactor needs to do

install_with_fresh_lockfile already produces everything install_frozen_lockfile's pipeline needs. After step 4 above (VirtualStoreLayout::new), the diff is:

• Replace step 5 (install_subtree) with:

  • compute_skipped_snapshots against built_lockfile.snapshots + the empty current_snapshots/current_packages (no prior install)
  • CreateVirtualStore::run with built_lockfile's packages / snapshots, the resolved layout, and the same PrefetchingResolver-populated MemCache the resolve phase already populated
  • SymlinkDirectDependencies::run with built_lockfile's importers
  • LinkVirtualStoreBins::run

• Plumb the install-pass state in: tarball_mem_cache, verified_files_cache, store_index, store_index_writer are already owned in install_with_fresh_lockfile; CreateVirtualStore::run already takes the same handles in install_frozen_lockfile. Just thread them through.

• Direct-dep tracking: today install_subtree is invoked once per direct dep via peers_result.direct_dependencies_by_alias. The refactored path needs to surface the same (alias, depPath) pairs to SymlinkDirectDependencies. direct_dependencies_by_alias is already built; built_lockfile.importers should already carry the importer-keyed direct list.

• Build phase: install_frozen_lockfile calls BuildModules after the link phase. install_with_fresh_lockfile should mirror that, gated by the same allow_build_policy.

• Hoisted linker: install_frozen_lockfile has a hoisted branch that dispatches through link_hoisted_modules instead of SymlinkDirectDependencies + LinkVirtualStoreBins. The refactored fresh-lockfile path needs to mirror this so nodeLinker: hoisted still works.

Expected impact

• full-resolution warm: 22.1 s → ~11–12 s (resolve 4–5 s + warm-batch link ~7 s, matching the frozen-warm path that lands at 7.5 s on the same fixture)
• clean-install cold: 25.5 s → ~18–20 s (cold path still pays the network + extract cost, but the link phase goes through the warm/cold batch and stops blocking tokio workers on rayon)
• frozen-lockfile scenarios: unchanged (already routed through CreateVirtualStore)

Net: closes the headline pnpm-vs-pacquet gap on the two scenarios where pnpm currently leads, without regressing the two where pacquet already leads.

Out of scope

• The resolve phase itself (pick_package, pick_package_from_meta, peer resolution) keeps its current cost. Profile shows 4-5 s on a 3 343-package fixture; that's not the headline gap and the existing PrefetchingResolver from perf(pacquet): close the clean-install gap to pnpm CLI #11856 already overlaps tarball downloads with resolution.
• The kernel-side clonefileat contention pattern from pacquet: pacquet install slower than pnpm install due to syscall contention #11851 is real on 16-core+ hosts but is a separate axis. None of the experiments in pacquet-perf5 moved the wall on the 10-core local box, and the architectural fix above is the one that matters across all hardware.

Plan

1. Lift install_frozen_lockfile's lockfile-driven pipeline (steps 5–7 above) into a shared helper that takes the lockfile shape + the install-scoped handles.
2. Wire install_with_fresh_lockfile to call that helper after build_fresh_lockfile + VirtualStoreLayout::new.
3. Delete install_subtree (and install_package_from_registry's caller side, if no other entry point remains).
4. Bench all four scenarios to confirm the wins and the no-regression on frozen.

Will track the PR against this issue.

Written by an agent (Claude Code, claude-opus-4-7).