Follow-up to #11866 / #11867. The architectural refactor in #11867 closed a small slice of the no-lockfile gap to pnpm (3–8% wall on ubuntu-latest), but the dominant remaining cost is in the resolve phase, not the install pipeline. This issue tracks the specific structural fix.
Evidence
Integrated-benchmark on ubuntu-latest, subtracting frozen-cold from the no-lockfile scenarios to isolate the resolve cost:
|
resolve wall |
resolve user CPU |
| pacquet@HEAD (#11867 applied) |
6.23 s |
7.83 s |
| pnpm |
1.80 s |
2.59 s |
Pacquet's resolve does ~3× more user CPU and ~3.5× more wall time than pnpm's for the same dep graph. sample(1) on the resolve window points at:
| frame |
samples |
resolve_peers::Walker::resolve_node |
532 |
resolve_dependency_tree::resolve_node |
353 |
pick_package_from_meta::{filter_pkg_metadata_by_publish_date, pick_package_from_meta} |
446 |
pick_package::{pick_package, pick_matching_version_fast, handle_cache_hit, pick_matching_version_final} |
421 |
npm_resolver::NpmResolver (trait impl) |
234 |
mirror::load_meta |
86 |
Tree-walking dominates. The picker family is hot too, but most of those calls are themselves redundant because the tree walker visits each package multiple times.
Root cause
pnpm's resolveDependencies.ts carries two memoisation caches on ResolutionContext (lines 160–164):
export interface ResolutionContext {
// ...
resolvedPkgsById: ResolvedPkgsById // pkgId → already-resolved package
childrenByParentId: ChildrenByParentId // parent pkgId → resolved child addresses
// ...
}
When pnpm reaches a package via the Nth parent edge, the isNew gate (line 1584) takes the early-out:
const isNew = !ctx.resolvedPkgsById[pkgResponse.body.id]
if (isNew) {
// heavy getResolvedPackage + recurse into children
ctx.resolvedPkgsById[id] = getResolvedPackage({ ... })
// (children resolution happens here, then cached as below)
ctx.childrenByParentId[parentPkg.pkgId] = pkgAddresses.map((child) => ({ ... }))
} else {
// just AND-fold the dev/prod/optional flags onto the cached envelope
ctx.resolvedPkgsById[id].prod = ... || !wantedDependency.dev && !wantedDependency.optional
ctx.resolvedPkgsById[id].dev = ... || wantedDependency.dev
ctx.resolvedPkgsById[id].optional = ... && currentIsOptional
}
The children walk for parentPkg.pkgId is materialised exactly once per unique parent; subsequent visits hit the childrenByParentId cache and don't recurse.
Pacquet's resolve_node (pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs:329) does not have this short-circuit. The dedup at packages.get_mut(&id) (line 393) only updates the optional flag on revisit — execution then falls through unconditionally:
// line 426-454
let is_leaf = pkg_is_leaf(&result);
let node_id = if is_leaf { NodeId::leaf(&id) } else { NodeId::next() };
// ...
let child_specs = extract_children(&result);
let child_results = child_specs
.into_iter()
.map(|(child_name, child_range, child_optional)| {
// ... build child WantedDependency ...
async move {
resolve_node(ctx, resolver, child_wanted, &next_ancestors, depth + 1, current_is_optional).await
}
})
.pipe(future::try_join_all)
.await?;
A package reached via N parent paths walks its entire subtree N times. With ~1 300 unique packages on the alotta-files fixture and an average of ~3 incoming paths per package, that's ~3× more resolve_node invocations than pnpm makes — matching the observed CPU ratio.
The picker hotspots (pick_package* family, 421 samples on top of pick_package_from_meta's 446) are largely downstream of this: each redundant resolve_node call invokes resolver.resolve(&wanted, ...) which goes through pick_package's in-memory packument cache but still does the per-pick CPU work — semver matching, mirror-path computation, format!-based cache-key construction. Eliminate the redundant visits and most of the picker churn disappears with them.
Proposed fix
Mirror pnpm's two-cache shape on TreeCtx:
resolved_subtree_node_id: Mutex<HashMap<String, NodeId>> — maps pkgIdWithPatchHash → the NodeId of the first occurrence's tree node. Later visitors reuse the same NodeId.
children_by_parent_id: Mutex<HashMap<String, Arc<Vec<(String, NodeId)>>>> — maps parent's pkgIdWithPatchHash → its resolved children list. First visitor builds the list; later visitors clone the Arc and stop.
resolve_node's control flow becomes:
let id = build_pkg_id_with_patch_hash(ctx, &result).await?;
// Cycle break (unchanged) ...
// AND-fold flags on the ResolvedPackage envelope (cheap, same as today).
update_resolved_package_flags(&ctx.packages, &id, current_is_optional, ...);
// Short-circuit: if we've already walked this id's subtree, reuse the NodeId
// and cached children — no recursion, no resolver chain reentry.
if let Some(node_id) = ctx.resolved_subtree_node_id.lock().get(&id).cloned() {
return Ok(Some(DirectDep { alias, node_id }));
}
let node_id = NodeId::next();
ctx.resolved_subtree_node_id.lock().insert(id.clone(), node_id.clone());
// First visit: walk children, build dependencies_tree node, cache children.
let child_specs = extract_children(&result);
let child_dep_paths = /* try_join_all child_specs → resolve_node */;
ctx.children_by_parent_id.lock().insert(id.clone(), Arc::new(child_dep_paths.clone()));
ctx.dependencies_tree.lock().insert(node_id, DependenciesTreeNode {
resolved_package_id: id,
children: child_dep_paths,
});
Ok(Some(DirectDep { alias, node_id }))
The peer-resolution pass (resolve_peers.rs) already keys on NodeId; reusing NodeIds across visits doesn't break it as long as pkg_is_leaf rules and the cycle-break logic stay intact. Verify with the existing tests in pacquet-resolving-deps-resolver — particularly the diamond and cycle scenarios.
Composing with the existing peer-suffix logic
The cached NodeId reuse needs to compose with pacquet's current is_leaf rule (line 426-427), which already returns a stable NodeId::leaf(&id) for leaf packages. Non-leaves currently get a fresh NodeId::next() per occurrence so peer-resolution can attach different peer suffixes per call site — but the peer suffixes are already a function of the parents in scope at peer-resolution time, not of the NodeId itself, so collapsing non-leaf visits onto a single NodeId should still produce the right peer variants.
This needs careful walking-through against the test suite. If a non-leaf collapse breaks peer-suffix variants, the fallback is to only cache childrenByParentId (skip the NodeId collapse) — the win is still substantial because the recursive subtree walk is the dominant cost.
Estimate
Closing 40–60% of the 6.2 s resolve wall on alotta-files. The picker churn (~870 samples) collapses with the redundant visits; the tree-walker work (~885 samples in resolve_node × 2) drops to roughly half. That would bring the no-lockfile clean-install wall down from 8.6 s towards ~5 s — roughly matching pnpm's 6.5 s, possibly beating it.
Out of scope
- The
pick_package_from_meta per-call Vec<&str> rebuild and Version::parse per candidate — a separate, smaller win worth doing afterward.
- Caching the parsed
node_semver::Version set on Package so max_satisfying doesn't reparse.
Object.create(preferredVersions) prototype-chain layering — pnpm's trick depends on JS semantics; a Rust equivalent (Cow<HashMap> or layered lookups) is a larger refactor with smaller payoff.
Validation plan
- Implement the two caches on
TreeCtx.
- Run
cargo nextest run -p pacquet-resolving-deps-resolver — the existing diamond / cycle / peer-variant tests are the safety net.
- Run
cargo nextest run -p pacquet-package-manager to catch any downstream graph-shape regressions.
- Bench against the
alotta-files fixture (integrated-benchmark, --scenario=clean-install and --scenario=full-resolution).
Written by an agent (Claude Code, claude-opus-4-7).
Follow-up to #11866 / #11867. The architectural refactor in #11867 closed a small slice of the no-lockfile gap to pnpm (3–8% wall on
ubuntu-latest), but the dominant remaining cost is in the resolve phase, not the install pipeline. This issue tracks the specific structural fix.Evidence
Integrated-benchmark on
ubuntu-latest, subtracting frozen-cold from the no-lockfile scenarios to isolate the resolve cost:Pacquet's resolve does ~3× more user CPU and ~3.5× more wall time than pnpm's for the same dep graph.
sample(1)on the resolve window points at:resolve_peers::Walker::resolve_noderesolve_dependency_tree::resolve_nodepick_package_from_meta::{filter_pkg_metadata_by_publish_date, pick_package_from_meta}pick_package::{pick_package, pick_matching_version_fast, handle_cache_hit, pick_matching_version_final}npm_resolver::NpmResolver(trait impl)mirror::load_metaTree-walking dominates. The picker family is hot too, but most of those calls are themselves redundant because the tree walker visits each package multiple times.
Root cause
pnpm's
resolveDependencies.tscarries two memoisation caches onResolutionContext(lines 160–164):When pnpm reaches a package via the Nth parent edge, the
isNewgate (line 1584) takes the early-out:The children walk for
parentPkg.pkgIdis materialised exactly once per unique parent; subsequent visits hit thechildrenByParentIdcache and don't recurse.Pacquet's
resolve_node(pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs:329) does not have this short-circuit. The dedup atpackages.get_mut(&id)(line 393) only updates theoptionalflag on revisit — execution then falls through unconditionally:A package reached via N parent paths walks its entire subtree N times. With ~1 300 unique packages on the
alotta-filesfixture and an average of ~3 incoming paths per package, that's ~3× moreresolve_nodeinvocations than pnpm makes — matching the observed CPU ratio.The picker hotspots (
pick_package*family, 421 samples on top ofpick_package_from_meta's 446) are largely downstream of this: each redundantresolve_nodecall invokesresolver.resolve(&wanted, ...)which goes throughpick_package's in-memory packument cache but still does the per-pick CPU work — semver matching, mirror-path computation,format!-based cache-key construction. Eliminate the redundant visits and most of the picker churn disappears with them.Proposed fix
Mirror pnpm's two-cache shape on
TreeCtx:resolved_subtree_node_id: Mutex<HashMap<String, NodeId>>— mapspkgIdWithPatchHash→ theNodeIdof the first occurrence's tree node. Later visitors reuse the sameNodeId.children_by_parent_id: Mutex<HashMap<String, Arc<Vec<(String, NodeId)>>>>— maps parent'spkgIdWithPatchHash→ its resolved children list. First visitor builds the list; later visitors clone theArcand stop.resolve_node's control flow becomes:The peer-resolution pass (
resolve_peers.rs) already keys onNodeId; reusingNodeIds across visits doesn't break it as long aspkg_is_leafrules and the cycle-break logic stay intact. Verify with the existing tests inpacquet-resolving-deps-resolver— particularly the diamond and cycle scenarios.Composing with the existing peer-suffix logic
The cached
NodeIdreuse needs to compose with pacquet's currentis_leafrule (line 426-427), which already returns a stableNodeId::leaf(&id)for leaf packages. Non-leaves currently get a freshNodeId::next()per occurrence so peer-resolution can attach different peer suffixes per call site — but the peer suffixes are already a function of the parents in scope at peer-resolution time, not of theNodeIditself, so collapsing non-leaf visits onto a singleNodeIdshould still produce the right peer variants.This needs careful walking-through against the test suite. If a non-leaf collapse breaks peer-suffix variants, the fallback is to only cache
childrenByParentId(skip theNodeIdcollapse) — the win is still substantial because the recursive subtree walk is the dominant cost.Estimate
Closing 40–60% of the 6.2 s resolve wall on
alotta-files. The picker churn (~870 samples) collapses with the redundant visits; the tree-walker work (~885 samples inresolve_node× 2) drops to roughly half. That would bring the no-lockfile clean-install wall down from 8.6 s towards ~5 s — roughly matching pnpm's 6.5 s, possibly beating it.Out of scope
pick_package_from_metaper-callVec<&str>rebuild andVersion::parseper candidate — a separate, smaller win worth doing afterward.node_semver::Versionset onPackagesomax_satisfyingdoesn't reparse.Object.create(preferredVersions)prototype-chain layering — pnpm's trick depends on JS semantics; a Rust equivalent (Cow<HashMap>or layered lookups) is a larger refactor with smaller payoff.Validation plan
TreeCtx.cargo nextest run -p pacquet-resolving-deps-resolver— the existing diamond / cycle / peer-variant tests are the safety net.cargo nextest run -p pacquet-package-managerto catch any downstream graph-shape regressions.alotta-filesfixture (integrated-benchmark,--scenario=clean-installand--scenario=full-resolution).Written by an agent (Claude Code, claude-opus-4-7).