Skip to content

perf(pacquet): close the warm-cache resolve gap to pnpm CLI#11837

Merged
zkochan merged 17 commits into
mainfrom
pacquet-perf
May 22, 2026
Merged

perf(pacquet): close the warm-cache resolve gap to pnpm CLI#11837
zkochan merged 17 commits into
mainfrom
pacquet-perf

Conversation

@zkochan

@zkochan zkochan commented May 21, 2026

Copy link
Copy Markdown
Member

Summary

Closes #11832.

On the alotta-files benchmark (1362 nodes, warm cache, GVS on), pacquet was 3-5× behind the TypeScript pnpm CLI whenever resolution ran (firstInstall, withWarmCache, withWarmModules, updatedDependencies). Wall-clock dropped from ~11.83s to ~5.03s on this branch; pnpm sits at ~4.16s, and the remaining gap is concentrated in the resolver's per-node pick_package walk (3.1s of the 5.03s — see #11843 for the peekManifestFromStore follow-up that would close it).

The branch is a series of small wins rather than one big rewrite. The original PrefetchingResolver (commit f375c91) was replaced by a batched store-index prefetch (461a4c0) — same throughput, far less plumbing.

What's in this PR

Resolve-phase

  • Packument fetch dedup (386a90b) — PackumentFetchLocker (per-cache-key DashMap<String, Arc<Semaphore>>) so concurrent picks of the same (registry, name) coalesce into one HTTP GET. Mirrors pnpm's runLimited(pkgMirror, …) in pickPackage.ts. Pacquet was firing N parallel GETs for the same packument per cluster of cross-referencing deps; now it's one.
  • Conditional GET on upgrade fetch (58f49c9) — forward etag / modified so the registry can answer 304 Not Modified on the abbreviated-to-full re-fetch path.
  • Off-tokio mirror disk reads (6cb50b4) — the packument cache's mirror read moves to spawn_blocking instead of running on the tokio worker.
  • Picked-manifest serialisation dedup (387b872) — PickedManifestCache Arc<DashMap<String, Arc<Value>>> so duplicate picks of the same name@version reuse the already-serialised Arc<Value> instead of re-running serde_json::to_value.
  • Arc-shared resolver outputs (743c718, 53e3cde, 5d6a420) — Package, ResolveResult.manifest, and ResolveResult itself are now shared via Arc so the tree walk's per-occurrence clones become refcount bumps.
  • std::sync::Mutex on TreeCtx (a7c94a9) — the per-package dedupe gate is a short HashMap insert with no await inside, so a sync mutex is the right tool. Tokio's async mutex was paying per-acquire overhead once per visit per ctx field on the resolve hot path.

Install-phase

  • Batched store-index prefetch (461a4c0) — one SELECT … WHERE key IN (…) against index.db at install start, rayon-parallel verify, drops the SQLite mutex before any fs work. Replaces the per-snapshot spawn_blocking fan-out that was serialising on Arc<Mutex<StoreIndex>> and queueing in tokio's blocking pool.
  • Single pnpm:progress per URL (e54208e) — run_with_mem_cache was emitting fetched twice when the in-memory cache hit; mirrors pnpm's packageRequester shape where the emit fires exactly once.
  • Retain prefetched manifests for bin linking (b0bf597) — the fresh-install bin linker now drives LinkVirtualStoreBins with the prefetched bundled-manifests map (and built lockfile snapshots), so per-child package.json disk reads on warm hits are gone. Skips the read_dir enumeration too. Updated snapshots reflect the now-present <slot>/node_modules/<pkg>/node_modules/.bin/<pkg> self-shim the lockfile-driven path writes per pnpm's linkBinsOfDependencies.
  • Skip per-snapshot deep clone on warm prefetch (3f9c1bb) — run_with_mem_cache returns the prefetched Arc<HashMap> straight through instead of going via run_without_mem_cache's deep-clone path + redundant Arc::new. At 1k+ snapshots that's one per-file map allocation and one Arc::new saved per snapshot.

Correctness

  • Registry-scoped PickedManifestCache key (57c3094) — the shared cache key was {name}@{version} only; two registries (default + JSR + named-registry) serving different artifacts under the same name@version (private + public collisions, forks) would hand one resolver the other's manifest. Now keyed {registry}\x00{name}@{version}, matching PackageMetaCache's shape. Regression test included.

Diagnostics

Review cleanup

  • Refactor + comment fixes (e7b3e6c) — build_resolve_result now takes a BuildResolveResult struct instead of 9 positional args (#[allow(clippy::too_many_arguments)] gone); resolver-side Arc bindings are dropped explicitly before the install pass so the packument cache actually frees; the resolve_dependency_tree doc comment was wrong about "skipping the recursion" on dedupe hits.

Doc + Dylint fixes

  • CI compliance (0343a47) — broken doc links resolved; single-letter generics, Arc.clone() direct, unicode ellipsis in doc comments fixed for Dylint.

Scope

  • Fresh-lockfile install path only. The frozen-lockfile path already had the batched store-index prefetch; nothing else here changes its behaviour.
  • No user-visible behavior change — lockfile format, error codes, CLI surface, MemCache semantics unchanged. The cache-key bug fix doesn't change the on-disk lockfile; it only prevents an in-memory mix-up that would otherwise produce a wrong ResolveResult.manifest field.

Follow-ups

  • perf(pacquet): port peekManifestFromStore fast path to skip the picker on hot-cache resolves #11843 — port pnpm's peekManifestFromStore fast path. The store-index row carries the bundled package.json (name, version, deps, bin, engines, etc.) but no publish-time, so the fast path is safe only when no published_by / minimumReleaseAge policy is in effect, no --update, and the wanted lockfile pins a tarball+integrity. With ~95% of nodes short-circuiting, the resolve_importer phase (currently 3.1s on warm cache) drops dramatically — this is the single biggest unimplemented win and the most likely path to parity with pnpm.

Benchmark

Wall-clock progression on the alotta-files warm-cache + GVS-on scenario (this branch vs main):

Stage Wall
main ~11.83s
+ packument fetch dedup ~8.21s
+ batched store-index prefetch ~6.39s
+ Arc-shared resolver outputs ~5.67s
+ std Mutex + manifest serial dedup ~5.03s
pnpm CLI baseline ~4.16s

The remaining ~0.87s gap is concentrated in resolve_importer and is what #11843 targets.

Test plan

  • cargo check --workspace --all-targets
  • cargo clippy --locked --workspace --all-targets -- --deny warnings
  • cargo fmt --all -- --check
  • cargo nextest run -p pacquet-package-manager — 298/298 pass
  • cargo nextest run -p pacquet-cli — 94/94 pass
  • cargo nextest run -p pacquet-tarball — 44/44 pass
  • cargo nextest run -p pacquet-resolving-npm-resolver — 245/245 pass (includes the shared_manifest_cache_does_not_leak_across_registries regression test for 57c3094)
  • cargo nextest run -p pacquet-resolving-deps-resolver — 28/28 pass
  • Re-run pnpm.io/benchmarks against this branch once it lands on pacquet-perf.

Written by an agent (Claude Code, claude-opus-4-7).

Pacquet currently resolves the entire dependency tree before any
tarball is fetched: `resolve_importer` walks every transitive dep
through the resolver chain and returns; only then does
`install_subtree` start calling `DownloadTarballToStore`. On the
`alotta-files` benchmark this puts pacquet 3-5x behind the
TypeScript pnpm CLI whenever resolution runs — pnpm
pipelines the two stages.

This change wraps the resolver chain in a `PrefetchingResolver`
that, after each successful resolve returning a tarball-shaped
result, `tokio::spawn`s a `DownloadTarballToStore::run_with_mem_cache`
into the shared `MemCache`. Resolution of children, siblings, and
the rest of the tree continues in parallel with that download.
When `install_subtree` later calls `run_with_mem_cache` for the
same URL, the per-URL cache either returns `CacheValue::Available`
immediately or briefly blocks on the existing `Notify`. Errors
surface to the install path as `TarballError::SiblingFetchFailed`,
unchanged.

Mirrors pnpm's `packageRequester.requestPackage` shape, which
returns a `pkgResponse` whose `fetching` field is a Promise that
is already running by the time the resolver returns
(`installing/package-requester/src/packageRequester.ts#L266`,
`installing/deps-resolver/src/resolveDependencies.ts#L1631`).

Mechanical changes to support the prefetch capture:
- `State::tarball_mem_cache` is now `Arc<MemCache>` so the spawned
  task can own a clone without leaking lifetimes.
- `store_index`, `store_index_writer`, and `verified_files_cache`
  are opened before the resolver chain instead of after
  `resolve_importer` so the prefetcher shares them with the
  install pass.
- `Reporter` bounds tightened to `+ 'static` along the path that
  feeds the spawn. All existing impls (`SilentReporter`,
  `NdjsonReporter`) already satisfy it.

Closes #11832.
@coderabbitai

coderabbitai Bot commented May 21, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Adds Arc-shared manifests and metadata, per-key packument fetch coalescing and picked-manifest cache, async mirror helpers and conditional-GET handling, recoverable std::sync::Mutex locking, Arc-owned tarball mem-cache with warm prefetching, and corresponding API/tests across resolver, npm-resolver, package-manager, CLI, and tarball crates.

Changes

Shared manifest ownership via Arc

Layer / File(s) Summary
Resolver base interface Arc wrapping
pacquet/crates/resolving-resolver-base/src/resolve.rs
Introduces SharedDependencyManifest = Arc<DependencyManifest>; ResolveResult.manifest and LatestInfo.latest_manifest become Option<Arc<...>>.
Engine runtime resolver implementations
pacquet/crates/engine-runtime-*/src/*_resolver.rs
Bun, Deno, and Node resolvers wrap generated JSON manifests in Arc before returning ResolveResult/LatestInfo.
Local resolver Arc wrapping
pacquet/crates/resolving-local-resolver/src/local_resolver.rs
LocalResolveResult.manifest changes to Option<Arc<serde_json::Value>>, directory-read manifests wrapped in Arc.
Dependencies graph and resolved package Arc wrapping
pacquet/crates/resolving-deps-resolver/*
DependenciesGraphNode.resolve_result and ResolvedPackage.result now store Arc<ResolveResult>; graph build and peer-resolution use Arc::clone.
Manifest borrowing adapter updates
pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs
Code using node.resolve_result.manifest switches from .as_ref() to .as_deref() for Arc-backed manifests.
Test updates for Arc wrapping
pacquet/crates/*/tests.rs
Test helpers and fixtures now construct/manipulate ResolveResult.manifest behind Arc where needed.

Resolver concurrency and locking refactor

Layer / File(s) Summary
Resolve dependency tree sync locking
pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs
Replace Tokio async mutexes with std::sync::Mutex and lock_recoverable helper; convert snapshot and resolved_versions to synchronous methods; wrap ResolveResult in Arc during resolution.
Resolve importer sync refactoring
pacquet/crates/resolving-deps-resolver/src/resolve_importer.rs
update_preferred_versions_with_ctx becomes synchronous; callers remove .await and use sync ctx.snapshot/resolved_versions.

Fetch coalescing and manifest caching infrastructure

Layer / File(s) Summary
Async mirror helpers and conditional HTTP fetch
pacquet/crates/resolving-npm-resolver/{mirror.rs,fetch_full_metadata*.rs,lib.rs}
Add load_meta_async and load_meta_headers_async; extend FetchFullMetadataOptions with etag/modified, add FetchFullMetadataOutcome::{Modified, NotModified}, and short-circuit on 304 responses.
Pick_package cache refactoring with Arc-metadata
pacquet/crates/resolving-npm-resolver/src/pick_package.rs
PackageMetaCache now stores Arc<Package>; introduce PackumentFetchLocker (DashMap<key, Arc>) and PickedManifestCache (Arc<DashMap<...>>); pick flow acquires per-key semaphore, re-checks cache, uses async mirror loads, and writes Arc-backed entries.
npm resolver cache and locker wiring
pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs
NpmResolver gains fetch_locker and picked_manifest_cache; PickedFromRegistry.meta becomes Arc<Package>; build_resolve_result uses picked_manifest_cache to cache serialized manifests as Arc<serde_json::Value>.
Named registry resolver cache wiring
pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs
Wires fetch_locker and picked_manifest_cache into pick/build flows.
Test suite cache wiring
pacquet/crates/resolving-npm-resolver/src/*/tests.rs, tests/chain.rs
Test harnesses now provide shared_packument_fetch_locker() and shared_picked_manifest_cache() and include a concurrency test asserting one GET for 20 concurrent picks.

Tarball cache ownership and install architecture

Layer / File(s) Summary
State Arc ownership
pacquet/crates/cli/src/state.rs
State.tarball_mem_cache changed to Arc<MemCache>; State::init constructs Arc::new(MemCache::new()).
CLI args Arc cache handling
pacquet/crates/cli/src/cli_args/{add.rs,install.rs}
AddArgs::run and InstallArgs::run tighten Reporter to ... + 'static and pass tarball_mem_cache via Arc::clone.
Add and Install struct Arc cache ownership
pacquet/crates/package-manager/src/{add.rs,install.rs}
Add and Install structs now own Arc<MemCache> instead of borrowing &MemCache; corresponding run generics tightened to 'static.
Fresh lockfile install Arc cache and prefetch infrastructure
pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs
InstallWithFreshLockfile holds Arc<MemCache>; creates shared fetch_locker and picked_manifest_cache, collects prefetch keys via collect_prefetch_cache_keys, runs batched prefetch_cas_paths, records prefetched_cas_paths, and threads them into per-package installers while adding timing/logging.
Per-package installation with prefetch threading
pacquet/crates/package-manager/src/install_package_from_registry.rs
InstallPackageFromRegistry gains prefetched_cas_paths: Option<&PrefetchedCasPaths> and uses resolution.manifest.as_deref() for unpacked_size; extract_tarball and manifest_unpacked_size made pub(crate).
Install and per-package test updates
pacquet/crates/package-manager/src/install/tests.rs, install_package_from_registry/tests.rs
Tests updated to use owned tarball_mem_cache: Default::default() and to include prefetched_cas_paths: None in struct literals; resolver test harnesses wired with shared cache/locker helpers.

Tarball cache progress cleanup

Layer / File(s) Summary
Tarball mem-cache progress cleanup
pacquet/crates/tarball/src/lib.rs
run_with_mem_cache no longer emits found_in_store progress events for subsequent requesters reusing an already-available mem-cache tarball.
Tarball progress test assertion update
pacquet/crates/tarball/src/tests.rs
Tests updated to assert the second requester does not emit FoundInStore when using run_with_mem_cache for the same URL.

Sequence Diagram(s)

sequenceDiagram
  participant Installer as InstallWithFreshLockfile
  participant Resolver as NpmResolver
  participant Picker as pick_package
  participant Locker as PackumentFetchLocker
  participant Registry as RegistryHTTP
  participant Store as DownloadTarballToStore

  Installer->>Resolver: resolve importer (build ResolveResult with manifest Arc)
  Resolver->>Picker: pick_package (meta: Arc<Package>)
  Picker->>Locker: acquire permit(cache_key)
  Locker-->>Picker: permit granted
  Picker->>Registry: conditional fetch (If-None-Match/If-Modified-Since)
  Registry-->>Picker: Modified / NotModified / 304
  Picker->>Store: start DownloadTarballToStore (returns fetching future)
  Installer->>Store: await fetching future during install_subtree
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related issues

  • #11842 — Implements Arc manifests, fetch coalescing, and warm-prefetch behavior that address the resolve-vs-fetch pipelining gap described in this issue.
  • #11756 — Adds resolver wiring and shared caches; this PR provides fetch_locker/picked_manifest_cache that align with that work.
  • #11841 — Modifies fetch_full_metadata and conditional GET handling, overlapping the same metadata-fetch paths.

Possibly related PRs

  • pnpm/pnpm#11783 — runtime resolver scaffold changes that the Arc-wrapping edits build upon.
  • pnpm/pnpm#11816 — lockfile conversion and dependencies_graph adaptations adjacent to Arc-manifest work.
  • pnpm/pnpm#11755 — prior resolver/picker work extended here with shared manifest caching and fetch coalescing.

"🐰 I wrapped manifests in Arc so they share,
I locked fetches so only one will spare,
Prefetch warms the store under sunlit art,
Mutexes recover and tests do their part,
Hopping installs now race and play — hop, hop, hooray! 🥕"

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR successfully implements the core objective from #11832: adopting pnpm's pipelined tarball-fetch model by making State::tarball_mem_cache shared (Arc), adding prefetch coordination with fetch_locker and picked_manifest_cache, wrapping manifests in Arc throughout the resolver chain, and enabling background tarball downloads during resolution via DownloadTarballToStore.
Out of Scope Changes check ✅ Passed All changes are scope-aligned: Arc-wrapping manifests in resolver outputs, adding fetch/manifest caches to NpmResolver and NamedRegistryResolver, supporting prefetched tarball paths in InstallPackageFromRegistry, and refactoring async mutexes to sync ones in resolve_dependency_tree are all necessary to pipeline tarball fetches with resolution without changing lockfile semantics or CLI behavior.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately summarizes the primary objective: closing performance gaps in tarball fetch resolution to match pnpm's pipelined approach through warm-cache prefetching.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch pacquet-perf

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov-commenter

codecov-commenter commented May 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.51%. Comparing base (a5a2c24) to head (e7b3e6c).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...package-manager/src/install_with_fresh_lockfile.rs 84.21% 15 Missing ⚠️
.../crates/resolving-npm-resolver/src/pick_package.rs 88.23% 10 Missing ⚠️
pacquet/crates/tarball/src/lib.rs 63.15% 7 Missing ⚠️
...es/engine-runtime-bun-resolver/src/bun_resolver.rs 0.00% 3 Missing ⚠️
.../engine-runtime-deno-resolver/src/deno_resolver.rs 0.00% 3 Missing ⚠️
.../engine-runtime-node-resolver/src/node_resolver.rs 0.00% 3 Missing ⚠️
...lving-deps-resolver/src/resolve_dependency_tree.rs 97.14% 1 Missing ⚠️
...es/resolving-deps-resolver/src/resolve_importer.rs 83.33% 1 Missing ⚠️
...ing-npm-resolver/src/fetch_full_metadata_cached.rs 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #11837      +/-   ##
==========================================
- Coverage   87.57%   87.51%   -0.06%     
==========================================
  Files         204      204              
  Lines       24134    24371     +237     
==========================================
+ Hits        21135    21329     +194     
- Misses       2999     3042      +43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Micro-Benchmark Results

Linux

group                          main                                   pr
-----                          ----                                   --
tarball/download_dependency    1.01      8.4±0.36ms   513.6 KB/sec    1.00      8.4±0.31ms   518.5 KB/sec

@github-actions

github-actions Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Integrated-Benchmark Report (Linux)

Scenario: Frozen Lockfile

Command Mean [s] Min [s] Max [s] Relative
pacquet@HEAD 2.606 ± 0.164 2.461 2.983 1.03 ± 0.07
pacquet@main 2.527 ± 0.056 2.464 2.620 1.00
pnpm 5.153 ± 0.046 5.042 5.192 2.04 ± 0.05
BENCHMARK_REPORT.json
{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 2.6055140832199997,
      "stddev": 0.16417504524716547,
      "median": 2.53837964412,
      "user": 2.8369985,
      "system": 3.8756796199999997,
      "min": 2.4612325656199996,
      "max": 2.98267009462,
      "times": [
        2.74438068162,
        2.98267009462,
        2.53370169462,
        2.71315596062,
        2.54305759362,
        2.4619441956199997,
        2.5233251546199997,
        2.59632646662,
        2.4953464246199997,
        2.4612325656199996
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 2.5270336008200003,
      "stddev": 0.055625847703279226,
      "median": 2.51127954662,
      "user": 2.8860426999999995,
      "system": 3.8362935200000003,
      "min": 2.46374816362,
      "max": 2.6202578056199997,
      "times": [
        2.4882527896199997,
        2.47512439362,
        2.61848091162,
        2.52093197762,
        2.6202578056199997,
        2.55987096562,
        2.50162711562,
        2.46374816362,
        2.52361787562,
        2.49842400962
      ]
    },
    {
      "command": "pnpm",
      "mean": 5.153018383619999,
      "stddev": 0.04624780026792434,
      "median": 5.16595848162,
      "user": 8.923904100000001,
      "system": 4.3675703200000005,
      "min": 5.04239478162,
      "max": 5.19171498362,
      "times": [
        5.16940291062,
        5.14812077362,
        5.19171498362,
        5.18572327762,
        5.17742640062,
        5.18967095862,
        5.15616921462,
        5.04239478162,
        5.16251405262,
        5.1070464826199995
      ]
    }
  ]
}

Scenario: Frozen Lockfile (Hot Cache)

Command Mean [ms] Min [ms] Max [ms] Relative
pacquet@HEAD 720.7 ± 30.7 692.3 772.8 1.02 ± 0.06
pacquet@main 708.4 ± 30.0 681.8 771.5 1.00
pnpm 2686.3 ± 93.3 2597.7 2934.0 3.79 ± 0.21
BENCHMARK_REPORT.json
{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 0.7207260035000002,
      "stddev": 0.0307457556462979,
      "median": 0.7103212955,
      "user": 0.41671423999999996,
      "system": 1.58135432,
      "min": 0.6923127635,
      "max": 0.7727502365000001,
      "times": [
        0.7727215855,
        0.7390981645,
        0.7150347015,
        0.7179170635000001,
        0.6929641865,
        0.7056078895000001,
        0.6977051185,
        0.7011483255000001,
        0.7727502365000001,
        0.6923127635
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 0.7084067916000001,
      "stddev": 0.03004218807885392,
      "median": 0.6926816575000001,
      "user": 0.40017084,
      "system": 1.5795114199999998,
      "min": 0.6817664105000001,
      "max": 0.7714558615,
      "times": [
        0.7714558615,
        0.6866672985000001,
        0.6890291285000001,
        0.7422537465000001,
        0.7149228235,
        0.6817664105000001,
        0.6854185585,
        0.7271907735,
        0.6890850255000001,
        0.6962782895
      ]
    },
    {
      "command": "pnpm",
      "mean": 2.6862958553000005,
      "stddev": 0.09333701720332611,
      "median": 2.6550140975000005,
      "user": 3.4177581399999992,
      "system": 2.28588132,
      "min": 2.5976800825,
      "max": 2.9339826225000003,
      "times": [
        2.7356068475,
        2.6653903205000002,
        2.6489391055000002,
        2.9339826225000003,
        2.6552308505,
        2.6738681185,
        2.5976800825,
        2.6468910085000004,
        2.6547973445000004,
        2.6505722525000004
      ]
    }
  ]
}

KSXGitHub pushed a commit that referenced this pull request May 21, 2026
Keep only the two new no-lockfile scenarios (`clean-install`,
`full-resolution`) on top of the existing `frozen-lockfile` and
`frozen-lockfile-hot-cache`. #11837's perf change is in the
fresh-lockfile install path, which only runs when resolution runs — i.e.,
exactly the no-lockfile scenarios. `peek` mutates an existing lockfile
and `gvs-warm` is a frozen-lockfile variant; neither exercises the
affected path, and including them only costs per-PR CI wall time.
Pacquet was firing N concurrent HTTP GETs for the same packument
whenever cross-referencing deps reached `pick_package` from
different points in the resolve tree (e.g. every `react-*` dep
racing for `react`). With no per-key serializer between the
in-memory cache miss check and the network fetch, all N concurrent
callers fell through to step 5 in parallel, queued behind the 16-64
slot `ThrottledClient` semaphore, and re-fetched the same packument
N times.

Upstream pnpm avoids this with `runLimited(pkgMirror, …)` — a
`pLimit(1)` keyed on the on-disk mirror path that wraps the entire
post-cache-miss flow. Only the first caller for a given packument
hits the network; the rest wait on the limit, then short-circuit on
the in-memory cache the winner just populated. See
`resolving/npm-resolver/src/pickPackage.ts:42-64`.

This change ports the same shape:

- `PackumentFetchLocker = Arc<DashMap<String, Arc<Semaphore>>>`
  threaded through `PickPackageContext::fetch_locker`. Constructed
  via `shared_packument_fetch_locker()` once per install and shared
  between `NpmResolver` and `NamedRegistryResolver` so concurrent
  picks across the two resolvers also coalesce.
- After step 1's in-memory cache miss check, `pick_package`
  acquires a single-permit semaphore keyed on the same string the
  cache uses (`{registry}\x00{name}[:full]`). After acquiring,
  re-checks the in-memory cache so a duplicate caller short-circuits
  on the winner's `meta_cache.set` instead of re-fetching.
- `handle_cache_hit` extracts the shared hit-then-upgrade-and-persist
  logic so the pre-permit and post-permit paths can't drift.

Test: `concurrent_picks_for_same_key_share_one_network_fetch` spawns
20 concurrent picks against a `mock.expect(1)` server. Without the
dedup, 20 GETs reach the registry; with it, exactly 1.

Closes #11832 (with the prefetch in the previous commit).
@KSXGitHub

Copy link
Copy Markdown
Contributor

Dylint CI error (for the AI agent to read):

    Checking pacquet-resolving-parse-wanted-dependency v0.0.1 (/home/runner/work/pnpm/pnpm/pacquet/crates/resolving-parse-wanted-dependency)
error: Unicode `…` (U+2026) in comment
   --> pacquet/crates/resolving-npm-resolver/src/pick_package.rs:391:32
    |
391 |     // [`runLimited(pkgMirror, …)`](https://github.com/pnpm/pnpm/blob/f657b5cb44/resolving/npm-resolver/src/pickPackage.ts#L52-L64)
    |                                ^ help: use ASCII `...` instead: `...`
    |
    = note: `-D perfectionist::unicode-ellipsis-in-comments` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(perfectionist::unicode_ellipsis_in_comments)]` 

@KSXGitHub

Copy link
Copy Markdown
Contributor

Doc CI error (for the AI agent to read):

 Documenting pacquet-engine-runtime-node-resolver v0.0.1 (/home/runner/work/pnpm/pnpm/pacquet/crates/engine-runtime-node-resolver)
error: public documentation for `tarball_mem_cache` links to private item `install_subtree`
  --> pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs:88:14
   |
88 |     /// in [`install_subtree`] still takes `&MemCache` via deref.
   |              ^^^^^^^^^^^^^^^ this item is private
   |
   = note: this link resolves only because you passed `--document-private-items`, but will break without
   = note: `-D rustdoc::private-intra-doc-links` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(rustdoc::private_intra_doc_links)]`

error: unresolved link to `TarballError::SiblingFetchFailed`
  --> pacquet/crates/package-manager/src/prefetching_resolver.rs:24:15
   |
24 | //! path as [`TarballError::SiblingFetchFailed`].
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no item named `TarballError` in scope
   |
   = note: `-D rustdoc::broken-intra-doc-links` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(rustdoc::broken_intra_doc_links)]`

error: could not document `pacquet-package-manager`
warning: build failed, waiting for other jobs to finish...

zkochan added 5 commits May 21, 2026 22:13
… fetch

When `minimumReleaseAge` is active, the resolver fetches abbreviated
metadata first and then upgrades to full metadata to read each
version's `time` field. Pacquet's upgrade fetch went out
unconditionally — no `If-None-Match` / `If-Modified-Since` — so
every triggered upgrade redownloaded the full packument body even
when the registry's full-form representation hadn't changed.

Upstream pnpm forwards `meta.etag` and `meta.modified` to the
upgrade fetch and treats `304 Not Modified` as "keep the
abbreviated meta and fall through to the picker's warn-and-skip
path" (`resolving/npm-resolver/src/pickPackage.ts:488-499`).

This change matches that:

- `FetchFullMetadataOptions` gains `etag` and `modified` fields.
- `fetch_full_metadata` honours them as conditional headers and
  returns a `FetchFullMetadataOutcome::{ Modified(Box<Package>),
  NotModified }` enum so callers can react to a 304.
- `maybe_upgrade_abbreviated_meta_for_release_age` forwards the
  abbreviated meta's etag/modified into the upgrade fetch and, on
  `NotModified`, returns the original meta untouched — same
  short-circuit as upstream's `notModified` arm.

Test: `fetch_full_metadata_returns_not_modified_on_304` asserts the
request carries the expected conditional headers and that a 304
response surfaces as `NotModified` without a body.

This is a correctness/parity fix; the user's benchmark showed it
doesn't move the wall-clock for their hot-cache scenario, which is
expected — once the abbreviated mirror has been written with the
upgraded full meta (carrying `time`), subsequent runs short-circuit
at the `meta.time.is_some()` guard before ever issuing the upgrade
fetch. The hot path is still the resolve walk itself; we'll keep
looking for the remaining gap separately.
…tokio worker

The pick-and-fetch path was calling synchronous `load_meta` and
`load_meta_headers` directly from async tasks. Each call reads the
mirror file (`fs::read_to_string`, a blocking syscall) and parses
the multi-KB to multi-MB packument body (`serde_json::from_str`,
CPU-bound). Neither yields, so the tokio worker stayed pinned for
the duration of every disk hit.

The hot path runs them on every cache-warm pick:
- `fetch_full_metadata_cached` calls `load_meta_headers` to feed
  the conditional GET headers.
- A 304 response then calls `load_meta` to re-materialise the
  packument.
- `pick_package`'s offline / pickLowestVersion, version-spec, and
  publishedBy shortcuts also call `load_meta` directly.

With ~600 unique packuments per install all running through this
sequence concurrently behind `try_join_all`, the blocking time
serializes against the worker pool size and the async scheduler
can't make progress on unrelated resolves / HTTP fetches in the
gaps.

This change adds async siblings — `load_meta_async` and
`load_meta_headers_async` — that dispatch the sync body to
`tokio::task::spawn_blocking`. Callers in `pick_package` and
`fetch_full_metadata_cached` await the async wrappers. The sync
functions stay (tests and the writeback path still call them
directly).

Matches upstream pnpm's posture, which performs the equivalent
work via awaited `fs.readFile` + `JSON.parse` on libuv's worker
pool.

Test: all 194 tests in `pacquet-resolving-npm-resolver` continue
to pass (existing `load_meta` / `load_meta_headers` coverage in
the mirror tests and via the fetcher integration tests is the
behaviour-level proof — the wrappers delegate verbatim).
`run_with_mem_cache`'s Available branches were re-emitting
`found_in_store` on every duplicate caller for an already-known
URL, so the per-package counter in pnpm's default-reporter ticked
by the number of resolve occurrences instead of by 1 — `~120×` on
the alotta-files benchmark, surfacing as a logged
`reused: 164746` for `1362` packages. The stderr write storm also
showed up directly as resolve-walk wall-clock.

Upstream's
[`packageRequester`](https://github.com/pnpm/pnpm/blob/086c5e91e8/installing/package-requester/src/packageRequester.ts#L410-L436)
attaches the emit via `.then()` on the first writer's promise;
later `await`s of the same promise do not re-trigger the handler.
This change matches that — only the first writer's
`run_without_mem_cache` call emits.

Inverts the existing
`mem_cache_hit_emits_found_in_store_for_second_requester` test to
`mem_cache_hit_does_not_re_emit_for_second_requester` — it now
asserts the second caller emits neither `found_in_store` nor
`fetched`.
…lockfile install

Under global virtual store the install side is essentially just
symlinking, so the resolve-walk wall-clock dominates — and almost
all of it ends up serialized on the `Arc<Mutex<StoreIndex>>`.

`run_with_mem_cache` → `run_without_mem_cache` calls
`load_cached_cas_paths` per package, which under the hood
`spawn_blocking`s a SQL `SELECT` against the shared store-index
mutex. When `PrefetchingResolver` fires that path once per resolve
occurrence (~1k spawns on the alotta-files fixture), the mutex
contention dominates wall-clock. `create_virtual_store::run`
already avoids this for the frozen-lockfile path via
`prefetch_cas_paths` — one batched `SELECT … WHERE key IN (…)`
followed by rayon-parallel per-row verify. This change ports that
shape to the fresh-lockfile path:

- After `resolve_importer`, walk the graph, collect every
  `(integrity, pkg_id)` cache key via the new
  `collect_prefetch_cache_keys` helper (mirrors the equivalent
  loop in `create_virtual_store::run`), run `prefetch_cas_paths`,
  and thread the resulting `PrefetchedCasPaths` through
  `InstallCtx` → `InstallPackageFromRegistry::run` →
  `DownloadTarballToStore` via the existing
  `prefetched_cas_paths` field.
- Drop the `PrefetchingResolver` wrapper. With the batched
  prefetch in place, the resolver chain is the plain
  `DefaultResolver` again — same setup the frozen-lockfile path
  uses.

Side effects:

- The `Resolved` → `found_in_store` → `Imported` emit ordering is
  fixed by construction: all three now fire from
  `InstallPackageFromRegistry::run` in that order. Previously the
  `PrefetchingResolver` spawn fired `found_in_store` from the
  resolve phase, before `Resolved` from the install phase.
- `prefetching_resolver.rs` is deleted.

Tests: 298/298 pass in `pacquet-package-manager`. Clippy clean.
…le install

To help target the remaining wall-clock gap, emit
`pacquet::install::phase` `tracing::info!`s at every major
boundary in `InstallWithFreshLockfile::run`:

- `resolve_importer`: deps-resolver walk
- `prefetch_cas_paths`: warm-cache batched lookup
- `build_fresh_lockfile`: graph → lockfile shape (GVS only)
- `virtual_store_layout_new`: per-snapshot GVS hash computation
- `install_subtree`: the per-package install fan-out

Carries the relevant counters (graph size, cache-key count, prefetch
hits) so a single trace tells us which phase dominates.

Run with `TRACE=pacquet::install::phase=info` to see the timings
without other noise.

No behavior change — purely observational.
zkochan added 5 commits May 21, 2026 23:48
…ache

Profiling under TRACE=pacquet::install::phase=info on the
alotta-files fixture put 8.2s of the 11.8s wall-clock inside
resolve_importer; the rest of the install pipeline totals ~1.9s.

`PackageMetaCache::get` was returning `Option<Package>` by value,
deep-cloning the full packument — including the `HashMap<String,
PackageVersion>` of every published version — on every hit.
Popular cross-referenced deps (`react`, `lodash`, …) get
requested 100+ times per install, so a single packument can be
deep-cloned 100+ times; with ~600 unique packuments the
allocations dominate the resolve walk.

JS Map semantics return references on `.get`, so pnpm's
metaCache pays zero copy on hits. This change matches that:

- `PackageMetaCache::get` returns `Option<Arc<Package>>`, `set`
  takes `Arc<Package>`.
- `InMemoryPackageMetaCache` stores `Arc<Package>`; the per-hit
  cost collapses to an `Arc::clone` (refcount bump).
- `PickPackageResult.meta`, `UpgradeOutcome.meta`, and
  `PickedFromRegistry.meta` become `Arc<Package>`. Callers that
  read fields keep working via `Arc`'s `Deref`; the places that
  needed an owned clone (cache writeback, persist-on-upgrade)
  now bump the `Arc` instead of cloning the body.

The `serde_json::to_value(picked)` call in `build_resolve_result`
still serialises one *version* into a JSON Value for the
resolver's `ResolveResult.manifest` field — that's a per-version
copy (≈KB), unavoidable until the resolver-base type takes the
typed `PackageVersion` directly.

Tests: 194/194 pass in `pacquet-resolving-npm-resolver`, 392/392
in `pacquet-package-manager` + `pacquet-cli`. Clippy clean.
…ia Arc

`ResolveResult` propagates by clone — once into `ResolvedPackage`
in the deps-resolver's `TreeCtx`, again into each
`DependenciesGraphNode` in the peer-resolved graph, and a few more
times through the install dispatch. The `manifest` field is a
`serde_json::Value` tree (per-version body) and every clone walked
it node-by-node.

Wrap it in `Arc` so `ResolveResult::clone()` bumps a refcount
instead of deep-cloning the JSON tree:

- New `SharedDependencyManifest = Arc<DependencyManifest>` alias on
  `pacquet-resolving-resolver-base`.
- `ResolveResult::manifest` and `LatestInfo::latest_manifest`
  re-typed as `Option<SharedDependencyManifest>`.
- All construction sites in npm / git / tarball / local / runtime
  resolvers and `resolve_from_workspace` wrap their built `Value`
  in `Arc::new`.
- Read sites mostly worked unchanged via `Arc<Value>`'s `Deref`
  coercion. Two call sites that pass `Option<&Value>` to a helper
  (`manifest_unpacked_size`, `build_package_metadata`) switched to
  `.as_deref()`.

Mirrors JS object-reference semantics — pnpm's `pkgResponse.manifest`
is an object, not a deep copy, so propagating the response across
resolve / install pays no copy.

Tests: 295/295 in the resolver crates pass; 298/298 in
`pacquet-package-manager`. Clippy clean.
…tion

`build_resolve_result` was calling `serde_json::to_value(picked)`
on every resolve to populate `ResolveResult::manifest`. For 1362
resolved occurrences with ~600 unique `(name, version)` pairs the
same picked `PackageVersion` was walked and reserialised hundreds
of times.

New `PickedManifestCache = Arc<DashMap<String, Arc<Value>>>`
(`shared_picked_manifest_cache()`) threaded through `NpmResolver`
and `NamedRegistryResolver`. On the first pick of a
`(name, version)` pair the resolver serialises once and stores
the `Arc<Value>` under the `{name}@{version}` key; subsequent
picks `Arc::clone` instead of reserialising.

The cache is install-scoped — built once in
`InstallWithFreshLockfile::run` and shared between the two
resolvers so cross-registry picks of the same package version
also coalesce.

Tests: 295/295 in the resolver crates pass; 492/492 across
`pacquet-resolving-npm-resolver` + `pacquet-package-manager`.
Clippy clean.
… on TreeCtx

`TreeCtx`'s state (`packages`, `dependencies_tree`,
`all_peer_dep_names`, `policy_violations`, `applied_patches`) was
guarded by `tokio::sync::Mutex`. Every visit acquired 2-3 of them
behind `.lock().await`, paying the async-mutex per-acquire
overhead (registration, future polling, wake) on the resolve hot
path.

None of the critical sections hold the lock across an `.await` —
they're short `HashMap` / `HashSet` inserts and `Vec::push`es.
That's exactly the shape `std::sync::Mutex` is for. Switch the
fields, remove the `.await`s, and route every acquire through a
new `lock_recoverable` helper that mirrors the rest of the
codebase's poisoning recovery (`build_modules.rs`,
`pick_package.rs`, …): an unrelated panic shouldn't escalate into
a hard install failure since the guarded values are plain
collections with no invariants that survive a panic.

Side effects:

- `TreeCtx::snapshot` and `TreeCtx::resolved_versions` drop the
  `async` qualifier and their callers drop the `.await`. The
  closures they live in are still `async` (they await elsewhere),
  so the change is mechanical.
- `update_preferred_versions_with_ctx` becomes sync along with
  `resolved_versions`.

Tests: 50/50 in `pacquet-resolving-deps-resolver` pass. Clippy
clean.
… tree + graph nodes

`ResolveResult.clone()` ran at every visit in `resolve_node`
(into `ResolvedPackage.result`) and again per
`(dep_path, peer-suffix)` slot when `resolve_peers` carved the
graph nodes out of the resolved tree (into
`DependenciesGraphNode.resolve_result`). Both clones deep-copied
every `String` field on `ResolveResult` (`id`, `alias`,
`resolved_via`, `name_ver`, etc.) — 2× per resolved package ×
~1.3k packages = ~2.7k full struct clones on the resolve hot
path.

Wrap once at the resolver boundary so both stores share one
heap allocation:

- `ResolvedPackage.result` and
  `DependenciesGraphNode.resolve_result` re-typed as
  `Arc<ResolveResult>`.
- `resolve_node` wraps the resolver-returned `ResolveResult` in
  `Arc::new` once and uses `Arc::clone` for the store + later
  reads. The `extract_children` / `extract_peer_dependencies`
  borrows work unchanged via `Arc`'s `Deref`.
- `resolve_peers`'s `.clone()` of `pkg.result` is now an
  `Arc::clone` (refcount bump), not a deep copy.

Read sites in `dependencies_graph_to_lockfile` and
`install_with_fresh_lockfile` work unchanged — `&Arc<T>` derefs
to `&T` at the field-init coercion site and at every
`.field`-style access.

Tests: 348/348 in `pacquet-resolving-deps-resolver` +
`pacquet-package-manager` pass. Clippy clean.
Doc job (`RUSTDOCFLAGS=-D warnings`):
- `cli/src/state.rs`: drop the doc-link to
  `pacquet_package_manager::PrefetchingResolver` (removed in the
  earlier batched-prefetch refactor); rewrite the comment to
  describe the current shape.
- `resolving-npm-resolver/src/pick_package.rs`: replace the
  unresolved `ResolveResult::manifest` link in `PickedManifestCache`'s
  doc with a fully-qualified path through `pacquet_resolving_resolver_base`.
- `resolving-npm-resolver/src/fetch_full_metadata.rs`: disambiguate
  the `crate::pick_package` link on `FetchFullMetadataOutcome` —
  the name resolves to both the function and the module, so route
  through `crate::pick_package()` and drop the bracketed link
  around the helper function name.

Dylint job (Perfectionist lints):
- `resolving-deps-resolver/src/resolve_dependency_tree.rs`: rename
  `lock_recoverable<T>` to `lock_recoverable<Inner>` so it doesn't
  trip `perfectionist::single-letter-generic`.
- `resolving-deps-resolver/src/resolve_peers.rs`: rewrite
  `pkg.result.clone()` (now `Arc<ResolveResult>`) as
  `Arc::clone(&pkg.result)` so the refcount bump is explicit at
  the call site, per `perfectionist::arc-rc-clone`.
- `resolving-npm-resolver/src/pick_package.rs`: replace the lone
  unicode `…` in the inline comment on the per-cache-key fetch
  serializer with ASCII `...`, per
  `perfectionist::unicode-ellipsis-in-comments`.
@zkochan zkochan marked this pull request as ready for review May 21, 2026 23:02
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs (1)

239-242: ⚡ Quick win

Add pre-assert logging for the non-assert_eq! check.

Line 239 uses assert!(matches!(...)) without a preceding log/debug line. Please add a quick dbg!(&outcome) (or equivalent) right before the assertion.

As per coding guidelines: "Log before non-assert_eq! assertions, dbg! complex structures, skip logging for simple scalar assert_eq!."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs`
around lines 239 - 242, Add a debug log right before the non-assert_eq!
assertion: insert a dbg!(&outcome) (or equivalent logging) immediately before
the assert!(matches!(outcome, FetchFullMetadataOutcome::NotModified)) line so
the complex `outcome` value is printed when the test fails; locate the assertion
using the symbol `outcome` and the pattern
`FetchFullMetadataOutcome::NotModified` in the test and place the dbg! call
immediately above it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs`:
- Around line 471-480: The current drop(resolver); drop(npm_resolver); doesn't
free the resolver's internal Arc caches because local Arcs (meta_cache,
fetch_locker, picked_manifest_cache) remain live; explicitly drop those Arc
bindings before the install pass instead of (or in addition to) dropping the
resolver wrapper. Locate the local variables meta_cache, fetch_locker, and
picked_manifest_cache in install_with_fresh_lockfile.rs and call
drop(meta_cache); drop(fetch_locker); drop(picked_manifest_cache); (or move them
out with std::mem::take/Option and then drop) immediately after dropping
resolver/npm_resolver so the resolver-side packument cache is actually released.

In `@pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs`:
- Around line 123-133: Update the comment to accurately reflect the control
flow: instead of saying later visitors “skip the recursion,” state that
concurrent visitors still build per-occurrence tree nodes and run child
resolution, but reuse the shared ResolvedPackage envelope stored in the shared
HashMap protected by the std::sync::Mutex; reference the use of try_join_all for
parallel sibling resolution, the shared HashMap/Mutex dedupe gate, the
ResolvedPackage envelope being reused, and that each visit still allocates a
fresh DependenciesTreeNode and resolves children (i.e., no short-circuit skip of
recursion).

In `@pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs`:
- Line 388: The introduced #[allow(clippy::too_many_arguments)] on
npm_resolver.rs should not be added without justification—either remove it and
refactor the affected function(s) to take a single options/context struct (e.g.,
create a NpmResolveOptions or ResolverContext type and replace the long
parameter list in the target function(s) and their call sites), or keep the
attribute but add an inline comment above it documenting the concrete, temporary
reason for the suppression and a TODO with a ticket/PR reference and target
milestone for the refactor; update the function signature(s) and all callers to
use the new struct if you choose refactoring.

In `@pacquet/crates/resolving-npm-resolver/src/pick_package.rs`:
- Around line 33-51: The doc comment contains Unicode ellipsis characters (“…”)
that Dylint rejects; replace all occurrences of “…” with an ASCII three-dot
sequence ("...") in the module-level comments and any inline docs in this file
(notably the comments describing PackumentFetchLocker,
PickPackageContext::fetch_locker and PackageMetaCache::set behavior and the
other doc-comment blocks around the packument/concurrency explanation),
including the other comment blocks referenced, so all doc comments use "..."
instead of the Unicode ellipsis.

---

Nitpick comments:
In `@pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs`:
- Around line 239-242: Add a debug log right before the non-assert_eq!
assertion: insert a dbg!(&outcome) (or equivalent logging) immediately before
the assert!(matches!(outcome, FetchFullMetadataOutcome::NotModified)) line so
the complex `outcome` value is printed when the test fails; locate the assertion
using the symbol `outcome` and the pattern
`FetchFullMetadataOutcome::NotModified` in the test and place the dbg! call
immediately above it.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 032d75e4-e790-441c-bd49-ebb5cf3f6d7a

📥 Commits

Reviewing files that changed from the base of the PR and between 9eb632b and 0343a47.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (40)
  • pacquet/crates/cli/src/cli_args/add.rs
  • pacquet/crates/cli/src/cli_args/install.rs
  • pacquet/crates/cli/src/state.rs
  • pacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rs
  • pacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rs
  • pacquet/crates/engine-runtime-node-resolver/src/node_resolver.rs
  • pacquet/crates/package-manager/src/add.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rs
  • pacquet/crates/package-manager/src/install.rs
  • pacquet/crates/package-manager/src/install/tests.rs
  • pacquet/crates/package-manager/src/install_package_from_registry.rs
  • pacquet/crates/package-manager/src/install_package_from_registry/tests.rs
  • pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs
  • pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs
  • pacquet/crates/resolving-deps-resolver/src/resolved_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/tests.rs
  • pacquet/crates/resolving-local-resolver/src/local_resolver.rs
  • pacquet/crates/resolving-npm-resolver/Cargo.toml
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rs
  • pacquet/crates/resolving-npm-resolver/src/lib.rs
  • pacquet/crates/resolving-npm-resolver/src/mirror.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rs
  • pacquet/crates/resolving-npm-resolver/tests/chain.rs
  • pacquet/crates/resolving-resolver-base/src/lib.rs
  • pacquet/crates/resolving-resolver-base/src/resolve.rs
  • pacquet/crates/tarball/src/lib.rs
  • pacquet/crates/tarball/src/tests.rs
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
  • GitHub Check: ubuntu-latest / Node.js 24 / Test
  • GitHub Check: Lint and Test (windows-latest)
  • GitHub Check: Lint and Test (macos-latest)
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Code Coverage
  • GitHub Check: Run benchmark on ubuntu-latest
🧰 Additional context used
📓 Path-based instructions (2)
pacquet/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

pacquet/**/*.rs: When porting a function that fires pnpm:<channel> events through globalLogger, logger.debug(), or streamParser.write(), mirror the call site, payload, and ordering so the reporter parses pacquet's NDJSON the same way it parses pnpm's.
Declare a newtype wrapper for branded string types. Do not collapse the brand into a plain String or &str.
If upstream always validates before construction, validate in pacquet's wrapper too. The wrapper must construct only via TryFrom<String> and/or FromStr. Do not provide an infallible public constructor.
If upstream never validates, just brand for type-safety. Expose an infallible From<String> (and From<&str> when convenient).
If upstream occasionally constructs without validation, expose from_str_unchecked as an escape hatch alongside the validating constructor.
Match upstream serde behavior for branded types that cross JSON, YAML, or INI boundaries. Use #[serde(try_from = "String")] for deserialization and #[serde(into = "String")] for serialization.
Use #[derive(derive_more::From)] and #[derive(derive_more::Into)] for mechanical conversion impls. Fall back to manual impl only when conversion needs custom logic.
String-literal unions should become enums, not newtype wrappers. Model closed sets of valid string values as enums.
Template literal types should be treated as branded strings with validation discipline from rules 2-5.
Choose owned vs. borrowed parameters to minimize copies. Widen to the most encompassing type (&Path over &PathBuf, &str over &String) when it doesn't force extra copies.
Prefer Arc::clone(&x) / Rc::clone(&x) over x.clone() for reference-counted types, so the cost is visible at the call site.
Follow Rust API Guidelines for naming conventions.
Do not use star imports inside module bodies. Write use super::{Foo, bar} instead of use super::*;. Two forms stay allowed: external-crate preludes like use rayon::prelude::*; and root-of-module re-...

Files:

  • pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rs
  • pacquet/crates/resolving-deps-resolver/src/resolved_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/mirror.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs
  • pacquet/crates/engine-runtime-node-resolver/src/node_resolver.rs
  • pacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rs
  • pacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rs
  • pacquet/crates/cli/src/cli_args/install.rs
  • pacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rs
  • pacquet/crates/resolving-npm-resolver/tests/chain.rs
  • pacquet/crates/resolving-npm-resolver/src/lib.rs
  • pacquet/crates/cli/src/cli_args/add.rs
  • pacquet/crates/tarball/src/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rs
  • pacquet/crates/resolving-local-resolver/src/local_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs
  • pacquet/crates/tarball/src/lib.rs
  • pacquet/crates/cli/src/state.rs
  • pacquet/crates/resolving-resolver-base/src/resolve.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rs
  • pacquet/crates/resolving-resolver-base/src/lib.rs
  • pacquet/crates/package-manager/src/install_package_from_registry.rs
  • pacquet/crates/package-manager/src/add.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rs
  • pacquet/crates/package-manager/src/install_package_from_registry/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rs
  • pacquet/crates/package-manager/src/install.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer.rs
  • pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package/tests.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package.rs
  • pacquet/crates/package-manager/src/install/tests.rs
pacquet/**/{tests,test}/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

pacquet/**/{tests,test}/**/*.rs: Prefer real fixtures; reach for the dependency-injection seam only when they can't cover the branch. Most happy paths and error paths should be tested with tempfile::TempDir, the mocked registry, or an integration test that spawns the actual binary.
Use the DI seam — a capability trait on the Host provider, threaded as Sys: <Bounds> — only for branches a real fixture can't reach portably: filesystem error kinds, deterministic time, shared process-global state, or external-service happy paths.
Log before non-assert_eq! assertions, dbg! complex structures, skip logging for simple scalar assert_eq!. Follow the test-logging guidance in CODE_STYLE_GUIDE.md.
Tests must not be tolerant of missing build/runtime environment by silently returning early when a tool isn't found. If the test needs a tool, let it panic when the tool is absent.
Prefer #[cfg_attr(target_os = "windows", ignore = "...")] (or #[cfg(unix)]) over a runtime probe-and-skip helper for platform-locked tools.
Port relevant tests from pnpm when porting behavior. Matching test coverage is the easiest way to prove behavioral parity.
When porting behavior from pnpm, consult plans/TEST_PORTING.md before adding ported tests. Use allow_known_failure! at the not-yet-implemented boundary and update checkboxes as items land.

Files:

  • pacquet/crates/resolving-npm-resolver/tests/chain.rs
🧠 Learnings (2)
📚 Learning: 2026-05-20T19:40:55.051Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11774
File: pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs:0-0
Timestamp: 2026-05-20T19:40:55.051Z
Learning: In the pacquet Rust code, ensure the semver implementation uses the `node-semver` crate (not `nodejs-semver`). `node-semver`’s public API does not include a `satisfies_with_prerelease`-style method; prerelease-tolerant matching should be implemented inline by first calling `Range::satisfies`, and when it rejects a prerelease version, retry matching against a stripped `MAJOR.MINOR.PATCH` base of the prerelease version.

Applied to files:

  • pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rs
  • pacquet/crates/resolving-deps-resolver/src/resolved_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/mirror.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs
  • pacquet/crates/engine-runtime-node-resolver/src/node_resolver.rs
  • pacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rs
  • pacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rs
  • pacquet/crates/cli/src/cli_args/install.rs
  • pacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rs
  • pacquet/crates/resolving-npm-resolver/tests/chain.rs
  • pacquet/crates/resolving-npm-resolver/src/lib.rs
  • pacquet/crates/cli/src/cli_args/add.rs
  • pacquet/crates/tarball/src/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rs
  • pacquet/crates/resolving-local-resolver/src/local_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs
  • pacquet/crates/tarball/src/lib.rs
  • pacquet/crates/cli/src/state.rs
  • pacquet/crates/resolving-resolver-base/src/resolve.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rs
  • pacquet/crates/resolving-resolver-base/src/lib.rs
  • pacquet/crates/package-manager/src/install_package_from_registry.rs
  • pacquet/crates/package-manager/src/add.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rs
  • pacquet/crates/package-manager/src/install_package_from_registry/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rs
  • pacquet/crates/package-manager/src/install.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer.rs
  • pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package/tests.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package.rs
  • pacquet/crates/package-manager/src/install/tests.rs
📚 Learning: 2026-05-20T23:07:58.444Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11784
File: pacquet/crates/resolving-deps-resolver/src/hoist_peers.rs:120-133
Timestamp: 2026-05-20T23:07:58.444Z
Learning: When reviewing code in this pacquet Rust port, follow the upstream pnpm compatibility rule: only match pnpm’s behavior exactly. Do not propose review changes that intentionally deviate from pnpm’s documented/observed behavior, even if pnpm appears buggy. If you identify a real bug in pnpm behavior, the review should prioritize fixing it upstream in pnpm first, and avoid implementing a pnpm-behavior workaround here unless the same fix has already landed upstream.

Applied to files:

  • pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rs
  • pacquet/crates/resolving-deps-resolver/src/resolved_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/mirror.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs
  • pacquet/crates/engine-runtime-node-resolver/src/node_resolver.rs
  • pacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rs
  • pacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rs
  • pacquet/crates/cli/src/cli_args/install.rs
  • pacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rs
  • pacquet/crates/resolving-npm-resolver/tests/chain.rs
  • pacquet/crates/resolving-npm-resolver/src/lib.rs
  • pacquet/crates/cli/src/cli_args/add.rs
  • pacquet/crates/tarball/src/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rs
  • pacquet/crates/resolving-local-resolver/src/local_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs
  • pacquet/crates/tarball/src/lib.rs
  • pacquet/crates/cli/src/state.rs
  • pacquet/crates/resolving-resolver-base/src/resolve.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rs
  • pacquet/crates/resolving-resolver-base/src/lib.rs
  • pacquet/crates/package-manager/src/install_package_from_registry.rs
  • pacquet/crates/package-manager/src/add.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rs
  • pacquet/crates/package-manager/src/install_package_from_registry/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rs
  • pacquet/crates/package-manager/src/install.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs
  • pacquet/crates/resolving-deps-resolver/src/resolve_importer.rs
  • pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package/tests.rs
  • pacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rs
  • pacquet/crates/resolving-npm-resolver/src/pick_package.rs
  • pacquet/crates/package-manager/src/install/tests.rs
🔇 Additional comments (33)
pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rs (1)

26-31: LGTM!

pacquet/crates/resolving-npm-resolver/Cargo.toml (1)

41-41: LGTM!

pacquet/crates/resolving-deps-resolver/src/resolved_tree.rs (1)

83-89: LGTM!

pacquet/crates/resolving-deps-resolver/src/tests.rs (1)

56-56: LGTM!

pacquet/crates/resolving-npm-resolver/src/mirror.rs (1)

235-272: LGTM!

pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs (1)

266-266: LGTM!

pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs (1)

34-34: LGTM!

Also applies to: 493-493

pacquet/crates/engine-runtime-node-resolver/src/node_resolver.rs (1)

158-158: LGTM!

Also applies to: 196-199

pacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rs (1)

112-112: LGTM!

Also applies to: 160-163

pacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rs (1)

119-119: LGTM!

Also applies to: 167-170

pacquet/crates/cli/src/cli_args/install.rs (1)

172-172: LGTM!

Also applies to: 229-229

pacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rs (1)

269-269: LGTM!

pacquet/crates/resolving-npm-resolver/tests/chain.rs (1)

25-25: LGTM!

Also applies to: 41-42

pacquet/crates/resolving-npm-resolver/src/lib.rs (1)

45-47: LGTM!

Also applies to: 62-65

pacquet/crates/cli/src/cli_args/add.rs (1)

93-96: LGTM!

Also applies to: 115-115

pacquet/crates/tarball/src/tests.rs (1)

1536-1542: LGTM!

Also applies to: 1549-1549, 1636-1642

pacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rs (1)

13-16: LGTM!

Also applies to: 74-75

pacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rs (1)

13-16: LGTM!

Also applies to: 66-67

pacquet/crates/resolving-local-resolver/src/local_resolver.rs (1)

79-79: LGTM!

Also applies to: 266-266

pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs (1)

72-80: LGTM!

Also applies to: 158-158, 213-213

pacquet/crates/tarball/src/lib.rs (1)

1782-1782: LGTM!

Also applies to: 1795-1801, 1817-1817

pacquet/crates/cli/src/state.rs (1)

10-10: LGTM!

Also applies to: 14-18, 75-75

pacquet/crates/resolving-resolver-base/src/resolve.rs (1)

11-11: LGTM!

Also applies to: 239-248, 274-277, 325-325

pacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rs (1)

55-55: LGTM!

pacquet/crates/resolving-resolver-base/src/lib.rs (1)

31-34: LGTM!

pacquet/crates/package-manager/src/install_package_from_registry.rs (1)

50-57: LGTM!

Also applies to: 118-118, 151-151, 180-180, 228-228, 255-255

pacquet/crates/package-manager/src/add.rs (1)

20-20: LGTM!

Also applies to: 54-54

pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rs (1)

30-31: LGTM!

Also applies to: 119-119, 151-153

pacquet/crates/package-manager/src/install_package_from_registry/tests.rs (1)

7-10: LGTM!

Also applies to: 103-105, 160-160, 246-246, 268-268, 360-360, 462-462

pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs (1)

96-107: LGTM!

Also applies to: 223-232, 270-279, 306-311, 381-385, 397-433

pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs (1)

3-15: LGTM!

Also applies to: 72-77, 104-105, 159-165, 194-195, 206-244

pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rs (1)

25-26: LGTM!

Also applies to: 40-42, 52-64, 66-82, 87-103, 112-143

pacquet/crates/package-manager/src/install/tests.rs (1)

52-52: LGTM!

Also applies to: 119-119, 190-190, 254-254, 335-335, 401-401, 487-487, 627-627, 726-726, 845-845, 966-966, 1062-1062, 1166-1166, 1214-1214, 1304-1304, 1396-1396, 1458-1458, 1521-1521, 1610-1610, 1715-1715, 1774-1774, 1860-1860, 1936-1936, 2018-2018, 2218-2218, 2340-2340, 2438-2438, 2542-2542, 2631-2631, 2722-2722, 2812-2812, 2898-2898, 2990-2990, 3080-3080, 3176-3176, 3274-3274, 3358-3358, 3440-3440, 3508-3508, 3572-3572, 3635-3635, 3701-3701, 3783-3783, 3843-3843, 3926-3926, 3975-3975, 4042-4042, 4112-4112, 4178-4178

Comment thread pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs
Comment thread pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs Outdated
Comment thread pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs Outdated
Comment thread pacquet/crates/resolving-npm-resolver/src/pick_package.rs
zkochan added 4 commits May 22, 2026 01:18
…by registry

`PickedManifestCache` is shared across the npm (default + JSR) and
named-registry resolvers, but its key was `{name}@{version}` only.
Two registries can serve different artifacts under the same
`name@version` — a public + private collision, or a fork — so a
collision would hand the second resolver the first resolver's
manifest, breaking the downstream dependency graph / peer
extraction / lockfile metadata.

Scope the key by registry (`{registry}\x00{name}@{version}`), matching
the existing `PackageMetaCache` shape, and add a regression test that
points two resolvers at different mock registries (each serving
distinct `dependencies` for `acme@1.0.0`) through one shared cache.
…install bin linking

The fresh-lockfile install path was passing `None` for both
`snapshots` and `package_manifests` to `LinkVirtualStoreBins`,
falling back to the readdir-driven `run_with_readdir` that walks
`<virtual_store_dir>` (or `<store_dir>/links` under GVS) and reads
each child's `package.json` from disk per slot. The
warm-cache prefetch already pulled every snapshot's bundled
manifest out of the SQLite store index ([`PrefetchResult::manifests`]),
so the disk reads were redundant.

Wire the prefetched manifests through:

- Destructure `PrefetchResult` so `manifests` survives past the
  `await` (was being dropped on `let prefetched_cas_paths =
  prefetch.cas_paths`).
- Always build the freshly-resolved lockfile structure (was gated
  on GVS); both branches now reuse one `built_lockfile`. The build
  is ~3 ms on the alotta-files fixture and is what we save below
  anyway.
- Project `prefetched_manifests` (keyed by
  `<integrity>\t<pkg_id>`) into a `PackageManifests` map
  (`PkgNameVerPeer.without_peer() → Arc<Value>`) by walking the
  resolved graph and computing the cache key per node — same key
  shape as `pacquet_store_dir::store_index_key`.
- Drive `LinkVirtualStoreBins` with `snapshots: Some(...)` +
  `package_manifests: &map`; cold-batch packages (cache miss →
  downloaded fresh in this run) fall through to a per-child disk
  read just like before.

`packages: None` on purpose: the freshly-built lockfile's
`packages:` rows carry an incomplete `has_bin` because the
resolver's `PackageVersion` deserializer does not include the
`bin` field. Trusting the empty-by-omission `has_bin_set` here
would filter out every child and skip bin linking entirely.
Once `bin` is threaded through `PackageVersion`, the call site
can switch to `built_lockfile.packages.as_ref()` to recover the
~95% slot short-circuit the frozen-lockfile path enjoys.

Updated snapshots reflect the now-present
`<slot>/node_modules/<pkg>/node_modules/.bin/<pkg>` self-shim
the lockfile-driven path writes per pnpm's
`linkBinsOfDependencies`. The readdir path skipped it via
`link_bins_excluding` — that was the divergence the existing
`same_global_virtual_store_layout_*` parity tests called out;
the fresh path now matches.
…fetched hits

`run_with_mem_cache` was delegating to `run_without_mem_cache`, which
ran the prefetched-cas-paths check and returned the hit via
`(**cas_paths).clone()` — a deep clone of the inner
`HashMap<String, PathBuf>`. `run_with_mem_cache` then wrapped the
result in `Arc::new(...)`. On a warm install with 1k+ snapshots
that is one full per-file map allocation per snapshot, plus an Arc
allocation, when the prefetched `Arc<HashMap<...>>` is already on
hand.

Lift the prefetched-cas-paths check into `run_with_mem_cache` so a
hit returns `Arc::clone(prefetched_entry)` straight through and
populates `mem_cache` with the same Arc — no per-entry walk, no
fresh `Arc::new`. The inner check inside `run_without_mem_cache`
stays for the direct callers (binary / zip archive paths in
`install_package_by_snapshot`) that still want an owned `HashMap`
they can mutate (e.g. to splice in the synthesized
`package.json` for runtime archives).
Three small cleanups flagged on #11837:

1. `install_with_fresh_lockfile.rs` — the `drop(resolver);
   drop(npm_resolver);` block didn't actually free the resolver-side
   caches because the local `meta_cache`, `fetch_locker`, and
   `picked_manifest_cache` `Arc` bindings each still held one strong
   reference of their own. Drop those bindings explicitly too so the
   cache shrinks before the install pass starts pulling tarballs.

2. `resolve_dependency_tree.rs` — the doc comment on
   `resolve_dependency_tree` said later visitors "skip the
   recursion," but the `Some(existing)` branch only AND-folds the
   `optional` flag and the code then unconditionally allocates a
   fresh `DependenciesTreeNode` and recurses on the children.
   Reword the comment to match what the code actually does (only
   the `ResolvedPackage` envelope is shared).

3. `npm_resolver.rs` — `build_resolve_result` had 9 positional
   arguments under `#[allow(clippy::too_many_arguments)]`. Drop the
   suppression and take a `BuildResolveResult` struct instead so the
   three call sites read as named-field literals and adding a future
   field doesn't require touching every caller.
@zkochan zkochan changed the title perf(pacquet): pipeline tarball fetches with resolution perf(pacquet): close the warm-cache resolve gap to pnpm CLI May 22, 2026
@zkochan zkochan merged commit 5353fcb into main May 22, 2026
28 checks passed
@zkochan zkochan deleted the pacquet-perf branch May 22, 2026 00:15
zkochan pushed a commit that referenced this pull request May 22, 2026
* ci(benchmark): run all six integrated-benchmark scenarios

Wires `clean-install`, `full-resolution`, `peek`, and `gvs-warm` into
`pacquet-integrated-benchmark.yml` so per-PR runs cover the same scenario
set the manual `benchmark.yml` workflow already exercises via
`benchmarks/bench.sh`. Requested for #11837, where the perf delta affects
the resolution-bound scenarios (`firstInstall`, `withWarmCache`,
`withWarmModules`, `updatedDependencies`) that the prior two-scenario set
did not measure.

Each scenario gets its own step with a 10 min hyperfine timeout (same
rationale as the existing steps) and writes per-scenario report copies
that the summary step concatenates into `SUMMARY.md`.

* ci(benchmark): drop peek and gvs-warm scenarios

Keep only the two new no-lockfile scenarios (`clean-install`,
`full-resolution`) on top of the existing `frozen-lockfile` and
`frozen-lockfile-hot-cache`. #11837's perf change is in the
fresh-lockfile install path, which only runs when resolution runs — i.e.,
exactly the no-lockfile scenarios. `peek` mutates an existing lockfile
and `gvs-warm` is a frozen-lockfile variant; neither exercises the
affected path, and including them only costs per-PR CI wall time.

* fix(bench): pin packages: ['.'] in synthesized pnpm-workspace.yaml

The integrated-benchmark clones each pacquet revision's source tree into
`<bench_dir>/pacquet/`, which on the pnpm/pnpm monorepo includes upstream
test fixtures like
`workspace/project-manifest-reader/__fixtures__/invalid-package-json/package.json`
— intentionally malformed JSON used to exercise pnpm's manifest reader.

Without a `packages:` field, both pnpm's `findPackages.ts:28` and
pacquet's `crates/workspace/src/projects.rs:128` default to `[".", "**"]`,
so the fresh-resolve install path's `find_workspace_projects` walk
descends into the cloned source tree and trips on the bad fixture:

  Error: pacquet_package_manifest::serialization_error
    × installing dependencies
    ╰─▶ expected `,` or `}` at line 3 column 3

The walk only runs on the fresh-lockfile branch (`install.rs:628-630`),
which is why frozen-lockfile and frozen-lockfile-hot-cache stay green
while clean-install and full-resolution fail every time.

Pin `packages: ['.']` in the synthesized manifest so enumeration stays
at the workspace root. The benchmark's installs are single-project,
so this doesn't narrow anything the install actually needed to see.
Fixtures supplied via `--fixture-dir` that already declare `packages:`
keep their own value.

* ci(benchmark): bump no-lockfile scenarios to 20 min

Clean-install and full-resolution go through pacquet's fresh-resolve
install path, which is currently ~3-5x slower than pnpm on the
`alotta-files` fixture (#11832). Hyperfine's default 1 warmup
+ 10 timed runs across three benchmark targets (pacquet@HEAD,
pacquet@main, system pnpm) projects to ~13 min wallclock for these
two scenarios, putting the previous 10 min cap right on the edge.
Doubling to 20 min keeps the per-step timeout meaningful as a stuck-
install detector without losing CI time when the bench is healthy.

The frozen-lockfile steps stay at 10 min — they don't traverse the
slower fresh-resolve path.

* fix(bench): drop --no-frozen-lockfile from full-resolution scenario

Pacquet doesn't expose `--no-frozen-lockfile` (only `--frozen-lockfile`,
`--prefer-frozen-lockfile`, and `--no-prefer-frozen-lockfile`). Passing
it makes clap reject the install:

  error: unexpected argument '--no-frozen-lockfile' found
    tip: a similar argument exists: '--frozen-lockfile'

The flag was redundant for this scenario anyway: full-resolution starts
every iteration with no lockfile on disk (init() skips the lockfile when
`lockfile_enabled()` is false; cleanup removes it; `lockfile=false` in
the synthesized npmrc/workspace prevents writing one). With no lockfile
present the frozen path is unreachable regardless of the flag, so both
tools take fresh resolution by definition. Fold full-resolution into
clean-install's bare `install` arm.

---------

Co-authored-by: Claude <noreply@anthropic.com>
KSXGitHub pushed a commit that referenced this pull request May 22, 2026
Resolved two conflicts against upstream changes in
`perf(pacquet): close the warm-cache resolve gap to pnpm CLI` (#11837)
and `fix(installing.deps-resolver): deterministically order cyclic peer
suffixes` (#11826):

- `package-manager/src/install_with_fresh_lockfile.rs`: main dropped the
  `enable_global_virtual_store` conditional and now builds the lockfile
  unconditionally (renamed `layout_lockfile` → `built_lockfile`, added a
  tracing span). The clippy `if_then_some_else_none` rewrite this branch
  added to that block is no longer applicable — took main's version.
- `resolving-deps-resolver/src/resolve_peers.rs` (two sites): main
  changed `NodeId` from a `Copy` newtype to an `enum { Counter(u64),
  Leaf(Arc<str>) }`, so `Some(node_id)` had to become
  `Some(node_id.clone())`. Combined with this branch's
  `if_then_some_else_none` rewrite of the sibling `alias` field. The new
  `node_id.clone()` does not trip `clippy::clone_on_ref_ptr` because
  `NodeId` is an enum, not an `Arc`/`Rc`/`Weak`.

`engine-runtime-node-resolver/src/node_resolver.rs` and
`resolving-npm-resolver/src/resolve_from_workspace.rs` auto-merged; the
clippy fixes on those lines survived intact.

Verified locally: `cargo clippy --locked --workspace --all-targets -- --deny warnings`, `cargo fmt --check`, `taplo format --check`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pacquet: pipeline tarball fetch with resolution to close the 3-5x gap on resolution-heavy installs

3 participants