perf(pacquet): close the warm-cache resolve gap to pnpm CLI#11837
Conversation
Pacquet currently resolves the entire dependency tree before any tarball is fetched: `resolve_importer` walks every transitive dep through the resolver chain and returns; only then does `install_subtree` start calling `DownloadTarballToStore`. On the `alotta-files` benchmark this puts pacquet 3-5x behind the TypeScript pnpm CLI whenever resolution runs — pnpm pipelines the two stages. This change wraps the resolver chain in a `PrefetchingResolver` that, after each successful resolve returning a tarball-shaped result, `tokio::spawn`s a `DownloadTarballToStore::run_with_mem_cache` into the shared `MemCache`. Resolution of children, siblings, and the rest of the tree continues in parallel with that download. When `install_subtree` later calls `run_with_mem_cache` for the same URL, the per-URL cache either returns `CacheValue::Available` immediately or briefly blocks on the existing `Notify`. Errors surface to the install path as `TarballError::SiblingFetchFailed`, unchanged. Mirrors pnpm's `packageRequester.requestPackage` shape, which returns a `pkgResponse` whose `fetching` field is a Promise that is already running by the time the resolver returns (`installing/package-requester/src/packageRequester.ts#L266`, `installing/deps-resolver/src/resolveDependencies.ts#L1631`). Mechanical changes to support the prefetch capture: - `State::tarball_mem_cache` is now `Arc<MemCache>` so the spawned task can own a clone without leaking lifetimes. - `store_index`, `store_index_writer`, and `verified_files_cache` are opened before the resolver chain instead of after `resolve_importer` so the prefetcher shares them with the install pass. - `Reporter` bounds tightened to `+ 'static` along the path that feeds the spawn. All existing impls (`SilentReporter`, `NdjsonReporter`) already satisfy it. Closes #11832.
📝 WalkthroughWalkthroughAdds Arc-shared manifests and metadata, per-key packument fetch coalescing and picked-manifest cache, async mirror helpers and conditional-GET handling, recoverable std::sync::Mutex locking, Arc-owned tarball mem-cache with warm prefetching, and corresponding API/tests across resolver, npm-resolver, package-manager, CLI, and tarball crates. ChangesShared manifest ownership via Arc
Resolver concurrency and locking refactor
Fetch coalescing and manifest caching infrastructure
Tarball cache ownership and install architecture
Tarball cache progress cleanup
Sequence Diagram(s)sequenceDiagram
participant Installer as InstallWithFreshLockfile
participant Resolver as NpmResolver
participant Picker as pick_package
participant Locker as PackumentFetchLocker
participant Registry as RegistryHTTP
participant Store as DownloadTarballToStore
Installer->>Resolver: resolve importer (build ResolveResult with manifest Arc)
Resolver->>Picker: pick_package (meta: Arc<Package>)
Picker->>Locker: acquire permit(cache_key)
Locker-->>Picker: permit granted
Picker->>Registry: conditional fetch (If-None-Match/If-Modified-Since)
Registry-->>Picker: Modified / NotModified / 304
Picker->>Store: start DownloadTarballToStore (returns fetching future)
Installer->>Store: await fetching future during install_subtree
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #11837 +/- ##
==========================================
- Coverage 87.57% 87.51% -0.06%
==========================================
Files 204 204
Lines 24134 24371 +237
==========================================
+ Hits 21135 21329 +194
- Misses 2999 3042 +43 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Micro-Benchmark ResultsLinux |
Integrated-Benchmark Report (Linux)Scenario: Frozen Lockfile
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 2.6055140832199997,
"stddev": 0.16417504524716547,
"median": 2.53837964412,
"user": 2.8369985,
"system": 3.8756796199999997,
"min": 2.4612325656199996,
"max": 2.98267009462,
"times": [
2.74438068162,
2.98267009462,
2.53370169462,
2.71315596062,
2.54305759362,
2.4619441956199997,
2.5233251546199997,
2.59632646662,
2.4953464246199997,
2.4612325656199996
]
},
{
"command": "pacquet@main",
"mean": 2.5270336008200003,
"stddev": 0.055625847703279226,
"median": 2.51127954662,
"user": 2.8860426999999995,
"system": 3.8362935200000003,
"min": 2.46374816362,
"max": 2.6202578056199997,
"times": [
2.4882527896199997,
2.47512439362,
2.61848091162,
2.52093197762,
2.6202578056199997,
2.55987096562,
2.50162711562,
2.46374816362,
2.52361787562,
2.49842400962
]
},
{
"command": "pnpm",
"mean": 5.153018383619999,
"stddev": 0.04624780026792434,
"median": 5.16595848162,
"user": 8.923904100000001,
"system": 4.3675703200000005,
"min": 5.04239478162,
"max": 5.19171498362,
"times": [
5.16940291062,
5.14812077362,
5.19171498362,
5.18572327762,
5.17742640062,
5.18967095862,
5.15616921462,
5.04239478162,
5.16251405262,
5.1070464826199995
]
}
]
}Scenario: Frozen Lockfile (Hot Cache)
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.7207260035000002,
"stddev": 0.0307457556462979,
"median": 0.7103212955,
"user": 0.41671423999999996,
"system": 1.58135432,
"min": 0.6923127635,
"max": 0.7727502365000001,
"times": [
0.7727215855,
0.7390981645,
0.7150347015,
0.7179170635000001,
0.6929641865,
0.7056078895000001,
0.6977051185,
0.7011483255000001,
0.7727502365000001,
0.6923127635
]
},
{
"command": "pacquet@main",
"mean": 0.7084067916000001,
"stddev": 0.03004218807885392,
"median": 0.6926816575000001,
"user": 0.40017084,
"system": 1.5795114199999998,
"min": 0.6817664105000001,
"max": 0.7714558615,
"times": [
0.7714558615,
0.6866672985000001,
0.6890291285000001,
0.7422537465000001,
0.7149228235,
0.6817664105000001,
0.6854185585,
0.7271907735,
0.6890850255000001,
0.6962782895
]
},
{
"command": "pnpm",
"mean": 2.6862958553000005,
"stddev": 0.09333701720332611,
"median": 2.6550140975000005,
"user": 3.4177581399999992,
"system": 2.28588132,
"min": 2.5976800825,
"max": 2.9339826225000003,
"times": [
2.7356068475,
2.6653903205000002,
2.6489391055000002,
2.9339826225000003,
2.6552308505,
2.6738681185,
2.5976800825,
2.6468910085000004,
2.6547973445000004,
2.6505722525000004
]
}
]
} |
Keep only the two new no-lockfile scenarios (`clean-install`, `full-resolution`) on top of the existing `frozen-lockfile` and `frozen-lockfile-hot-cache`. #11837's perf change is in the fresh-lockfile install path, which only runs when resolution runs — i.e., exactly the no-lockfile scenarios. `peek` mutates an existing lockfile and `gvs-warm` is a frozen-lockfile variant; neither exercises the affected path, and including them only costs per-PR CI wall time.
Pacquet was firing N concurrent HTTP GETs for the same packument
whenever cross-referencing deps reached `pick_package` from
different points in the resolve tree (e.g. every `react-*` dep
racing for `react`). With no per-key serializer between the
in-memory cache miss check and the network fetch, all N concurrent
callers fell through to step 5 in parallel, queued behind the 16-64
slot `ThrottledClient` semaphore, and re-fetched the same packument
N times.
Upstream pnpm avoids this with `runLimited(pkgMirror, …)` — a
`pLimit(1)` keyed on the on-disk mirror path that wraps the entire
post-cache-miss flow. Only the first caller for a given packument
hits the network; the rest wait on the limit, then short-circuit on
the in-memory cache the winner just populated. See
`resolving/npm-resolver/src/pickPackage.ts:42-64`.
This change ports the same shape:
- `PackumentFetchLocker = Arc<DashMap<String, Arc<Semaphore>>>`
threaded through `PickPackageContext::fetch_locker`. Constructed
via `shared_packument_fetch_locker()` once per install and shared
between `NpmResolver` and `NamedRegistryResolver` so concurrent
picks across the two resolvers also coalesce.
- After step 1's in-memory cache miss check, `pick_package`
acquires a single-permit semaphore keyed on the same string the
cache uses (`{registry}\x00{name}[:full]`). After acquiring,
re-checks the in-memory cache so a duplicate caller short-circuits
on the winner's `meta_cache.set` instead of re-fetching.
- `handle_cache_hit` extracts the shared hit-then-upgrade-and-persist
logic so the pre-permit and post-permit paths can't drift.
Test: `concurrent_picks_for_same_key_share_one_network_fetch` spawns
20 concurrent picks against a `mock.expect(1)` server. Without the
dedup, 20 GETs reach the registry; with it, exactly 1.
Closes #11832 (with the prefetch in the previous commit).
|
Dylint CI error (for the AI agent to read): |
|
Doc CI error (for the AI agent to read): |
… fetch
When `minimumReleaseAge` is active, the resolver fetches abbreviated
metadata first and then upgrades to full metadata to read each
version's `time` field. Pacquet's upgrade fetch went out
unconditionally — no `If-None-Match` / `If-Modified-Since` — so
every triggered upgrade redownloaded the full packument body even
when the registry's full-form representation hadn't changed.
Upstream pnpm forwards `meta.etag` and `meta.modified` to the
upgrade fetch and treats `304 Not Modified` as "keep the
abbreviated meta and fall through to the picker's warn-and-skip
path" (`resolving/npm-resolver/src/pickPackage.ts:488-499`).
This change matches that:
- `FetchFullMetadataOptions` gains `etag` and `modified` fields.
- `fetch_full_metadata` honours them as conditional headers and
returns a `FetchFullMetadataOutcome::{ Modified(Box<Package>),
NotModified }` enum so callers can react to a 304.
- `maybe_upgrade_abbreviated_meta_for_release_age` forwards the
abbreviated meta's etag/modified into the upgrade fetch and, on
`NotModified`, returns the original meta untouched — same
short-circuit as upstream's `notModified` arm.
Test: `fetch_full_metadata_returns_not_modified_on_304` asserts the
request carries the expected conditional headers and that a 304
response surfaces as `NotModified` without a body.
This is a correctness/parity fix; the user's benchmark showed it
doesn't move the wall-clock for their hot-cache scenario, which is
expected — once the abbreviated mirror has been written with the
upgraded full meta (carrying `time`), subsequent runs short-circuit
at the `meta.time.is_some()` guard before ever issuing the upgrade
fetch. The hot path is still the resolve walk itself; we'll keep
looking for the remaining gap separately.
…tokio worker The pick-and-fetch path was calling synchronous `load_meta` and `load_meta_headers` directly from async tasks. Each call reads the mirror file (`fs::read_to_string`, a blocking syscall) and parses the multi-KB to multi-MB packument body (`serde_json::from_str`, CPU-bound). Neither yields, so the tokio worker stayed pinned for the duration of every disk hit. The hot path runs them on every cache-warm pick: - `fetch_full_metadata_cached` calls `load_meta_headers` to feed the conditional GET headers. - A 304 response then calls `load_meta` to re-materialise the packument. - `pick_package`'s offline / pickLowestVersion, version-spec, and publishedBy shortcuts also call `load_meta` directly. With ~600 unique packuments per install all running through this sequence concurrently behind `try_join_all`, the blocking time serializes against the worker pool size and the async scheduler can't make progress on unrelated resolves / HTTP fetches in the gaps. This change adds async siblings — `load_meta_async` and `load_meta_headers_async` — that dispatch the sync body to `tokio::task::spawn_blocking`. Callers in `pick_package` and `fetch_full_metadata_cached` await the async wrappers. The sync functions stay (tests and the writeback path still call them directly). Matches upstream pnpm's posture, which performs the equivalent work via awaited `fs.readFile` + `JSON.parse` on libuv's worker pool. Test: all 194 tests in `pacquet-resolving-npm-resolver` continue to pass (existing `load_meta` / `load_meta_headers` coverage in the mirror tests and via the fetcher integration tests is the behaviour-level proof — the wrappers delegate verbatim).
`run_with_mem_cache`'s Available branches were re-emitting `found_in_store` on every duplicate caller for an already-known URL, so the per-package counter in pnpm's default-reporter ticked by the number of resolve occurrences instead of by 1 — `~120×` on the alotta-files benchmark, surfacing as a logged `reused: 164746` for `1362` packages. The stderr write storm also showed up directly as resolve-walk wall-clock. Upstream's [`packageRequester`](https://github.com/pnpm/pnpm/blob/086c5e91e8/installing/package-requester/src/packageRequester.ts#L410-L436) attaches the emit via `.then()` on the first writer's promise; later `await`s of the same promise do not re-trigger the handler. This change matches that — only the first writer's `run_without_mem_cache` call emits. Inverts the existing `mem_cache_hit_emits_found_in_store_for_second_requester` test to `mem_cache_hit_does_not_re_emit_for_second_requester` — it now asserts the second caller emits neither `found_in_store` nor `fetched`.
…lockfile install Under global virtual store the install side is essentially just symlinking, so the resolve-walk wall-clock dominates — and almost all of it ends up serialized on the `Arc<Mutex<StoreIndex>>`. `run_with_mem_cache` → `run_without_mem_cache` calls `load_cached_cas_paths` per package, which under the hood `spawn_blocking`s a SQL `SELECT` against the shared store-index mutex. When `PrefetchingResolver` fires that path once per resolve occurrence (~1k spawns on the alotta-files fixture), the mutex contention dominates wall-clock. `create_virtual_store::run` already avoids this for the frozen-lockfile path via `prefetch_cas_paths` — one batched `SELECT … WHERE key IN (…)` followed by rayon-parallel per-row verify. This change ports that shape to the fresh-lockfile path: - After `resolve_importer`, walk the graph, collect every `(integrity, pkg_id)` cache key via the new `collect_prefetch_cache_keys` helper (mirrors the equivalent loop in `create_virtual_store::run`), run `prefetch_cas_paths`, and thread the resulting `PrefetchedCasPaths` through `InstallCtx` → `InstallPackageFromRegistry::run` → `DownloadTarballToStore` via the existing `prefetched_cas_paths` field. - Drop the `PrefetchingResolver` wrapper. With the batched prefetch in place, the resolver chain is the plain `DefaultResolver` again — same setup the frozen-lockfile path uses. Side effects: - The `Resolved` → `found_in_store` → `Imported` emit ordering is fixed by construction: all three now fire from `InstallPackageFromRegistry::run` in that order. Previously the `PrefetchingResolver` spawn fired `found_in_store` from the resolve phase, before `Resolved` from the install phase. - `prefetching_resolver.rs` is deleted. Tests: 298/298 pass in `pacquet-package-manager`. Clippy clean.
…le install To help target the remaining wall-clock gap, emit `pacquet::install::phase` `tracing::info!`s at every major boundary in `InstallWithFreshLockfile::run`: - `resolve_importer`: deps-resolver walk - `prefetch_cas_paths`: warm-cache batched lookup - `build_fresh_lockfile`: graph → lockfile shape (GVS only) - `virtual_store_layout_new`: per-snapshot GVS hash computation - `install_subtree`: the per-package install fan-out Carries the relevant counters (graph size, cache-key count, prefetch hits) so a single trace tells us which phase dominates. Run with `TRACE=pacquet::install::phase=info` to see the timings without other noise. No behavior change — purely observational.
…ache Profiling under TRACE=pacquet::install::phase=info on the alotta-files fixture put 8.2s of the 11.8s wall-clock inside resolve_importer; the rest of the install pipeline totals ~1.9s. `PackageMetaCache::get` was returning `Option<Package>` by value, deep-cloning the full packument — including the `HashMap<String, PackageVersion>` of every published version — on every hit. Popular cross-referenced deps (`react`, `lodash`, …) get requested 100+ times per install, so a single packument can be deep-cloned 100+ times; with ~600 unique packuments the allocations dominate the resolve walk. JS Map semantics return references on `.get`, so pnpm's metaCache pays zero copy on hits. This change matches that: - `PackageMetaCache::get` returns `Option<Arc<Package>>`, `set` takes `Arc<Package>`. - `InMemoryPackageMetaCache` stores `Arc<Package>`; the per-hit cost collapses to an `Arc::clone` (refcount bump). - `PickPackageResult.meta`, `UpgradeOutcome.meta`, and `PickedFromRegistry.meta` become `Arc<Package>`. Callers that read fields keep working via `Arc`'s `Deref`; the places that needed an owned clone (cache writeback, persist-on-upgrade) now bump the `Arc` instead of cloning the body. The `serde_json::to_value(picked)` call in `build_resolve_result` still serialises one *version* into a JSON Value for the resolver's `ResolveResult.manifest` field — that's a per-version copy (≈KB), unavoidable until the resolver-base type takes the typed `PackageVersion` directly. Tests: 194/194 pass in `pacquet-resolving-npm-resolver`, 392/392 in `pacquet-package-manager` + `pacquet-cli`. Clippy clean.
…ia Arc `ResolveResult` propagates by clone — once into `ResolvedPackage` in the deps-resolver's `TreeCtx`, again into each `DependenciesGraphNode` in the peer-resolved graph, and a few more times through the install dispatch. The `manifest` field is a `serde_json::Value` tree (per-version body) and every clone walked it node-by-node. Wrap it in `Arc` so `ResolveResult::clone()` bumps a refcount instead of deep-cloning the JSON tree: - New `SharedDependencyManifest = Arc<DependencyManifest>` alias on `pacquet-resolving-resolver-base`. - `ResolveResult::manifest` and `LatestInfo::latest_manifest` re-typed as `Option<SharedDependencyManifest>`. - All construction sites in npm / git / tarball / local / runtime resolvers and `resolve_from_workspace` wrap their built `Value` in `Arc::new`. - Read sites mostly worked unchanged via `Arc<Value>`'s `Deref` coercion. Two call sites that pass `Option<&Value>` to a helper (`manifest_unpacked_size`, `build_package_metadata`) switched to `.as_deref()`. Mirrors JS object-reference semantics — pnpm's `pkgResponse.manifest` is an object, not a deep copy, so propagating the response across resolve / install pays no copy. Tests: 295/295 in the resolver crates pass; 298/298 in `pacquet-package-manager`. Clippy clean.
…tion
`build_resolve_result` was calling `serde_json::to_value(picked)`
on every resolve to populate `ResolveResult::manifest`. For 1362
resolved occurrences with ~600 unique `(name, version)` pairs the
same picked `PackageVersion` was walked and reserialised hundreds
of times.
New `PickedManifestCache = Arc<DashMap<String, Arc<Value>>>`
(`shared_picked_manifest_cache()`) threaded through `NpmResolver`
and `NamedRegistryResolver`. On the first pick of a
`(name, version)` pair the resolver serialises once and stores
the `Arc<Value>` under the `{name}@{version}` key; subsequent
picks `Arc::clone` instead of reserialising.
The cache is install-scoped — built once in
`InstallWithFreshLockfile::run` and shared between the two
resolvers so cross-registry picks of the same package version
also coalesce.
Tests: 295/295 in the resolver crates pass; 492/492 across
`pacquet-resolving-npm-resolver` + `pacquet-package-manager`.
Clippy clean.
… on TreeCtx `TreeCtx`'s state (`packages`, `dependencies_tree`, `all_peer_dep_names`, `policy_violations`, `applied_patches`) was guarded by `tokio::sync::Mutex`. Every visit acquired 2-3 of them behind `.lock().await`, paying the async-mutex per-acquire overhead (registration, future polling, wake) on the resolve hot path. None of the critical sections hold the lock across an `.await` — they're short `HashMap` / `HashSet` inserts and `Vec::push`es. That's exactly the shape `std::sync::Mutex` is for. Switch the fields, remove the `.await`s, and route every acquire through a new `lock_recoverable` helper that mirrors the rest of the codebase's poisoning recovery (`build_modules.rs`, `pick_package.rs`, …): an unrelated panic shouldn't escalate into a hard install failure since the guarded values are plain collections with no invariants that survive a panic. Side effects: - `TreeCtx::snapshot` and `TreeCtx::resolved_versions` drop the `async` qualifier and their callers drop the `.await`. The closures they live in are still `async` (they await elsewhere), so the change is mechanical. - `update_preferred_versions_with_ctx` becomes sync along with `resolved_versions`. Tests: 50/50 in `pacquet-resolving-deps-resolver` pass. Clippy clean.
… tree + graph nodes `ResolveResult.clone()` ran at every visit in `resolve_node` (into `ResolvedPackage.result`) and again per `(dep_path, peer-suffix)` slot when `resolve_peers` carved the graph nodes out of the resolved tree (into `DependenciesGraphNode.resolve_result`). Both clones deep-copied every `String` field on `ResolveResult` (`id`, `alias`, `resolved_via`, `name_ver`, etc.) — 2× per resolved package × ~1.3k packages = ~2.7k full struct clones on the resolve hot path. Wrap once at the resolver boundary so both stores share one heap allocation: - `ResolvedPackage.result` and `DependenciesGraphNode.resolve_result` re-typed as `Arc<ResolveResult>`. - `resolve_node` wraps the resolver-returned `ResolveResult` in `Arc::new` once and uses `Arc::clone` for the store + later reads. The `extract_children` / `extract_peer_dependencies` borrows work unchanged via `Arc`'s `Deref`. - `resolve_peers`'s `.clone()` of `pkg.result` is now an `Arc::clone` (refcount bump), not a deep copy. Read sites in `dependencies_graph_to_lockfile` and `install_with_fresh_lockfile` work unchanged — `&Arc<T>` derefs to `&T` at the field-init coercion site and at every `.field`-style access. Tests: 348/348 in `pacquet-resolving-deps-resolver` + `pacquet-package-manager` pass. Clippy clean.
Doc job (`RUSTDOCFLAGS=-D warnings`): - `cli/src/state.rs`: drop the doc-link to `pacquet_package_manager::PrefetchingResolver` (removed in the earlier batched-prefetch refactor); rewrite the comment to describe the current shape. - `resolving-npm-resolver/src/pick_package.rs`: replace the unresolved `ResolveResult::manifest` link in `PickedManifestCache`'s doc with a fully-qualified path through `pacquet_resolving_resolver_base`. - `resolving-npm-resolver/src/fetch_full_metadata.rs`: disambiguate the `crate::pick_package` link on `FetchFullMetadataOutcome` — the name resolves to both the function and the module, so route through `crate::pick_package()` and drop the bracketed link around the helper function name. Dylint job (Perfectionist lints): - `resolving-deps-resolver/src/resolve_dependency_tree.rs`: rename `lock_recoverable<T>` to `lock_recoverable<Inner>` so it doesn't trip `perfectionist::single-letter-generic`. - `resolving-deps-resolver/src/resolve_peers.rs`: rewrite `pkg.result.clone()` (now `Arc<ResolveResult>`) as `Arc::clone(&pkg.result)` so the refcount bump is explicit at the call site, per `perfectionist::arc-rc-clone`. - `resolving-npm-resolver/src/pick_package.rs`: replace the lone unicode `…` in the inline comment on the per-cache-key fetch serializer with ASCII `...`, per `perfectionist::unicode-ellipsis-in-comments`.
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs (1)
239-242: ⚡ Quick winAdd pre-assert logging for the non-
assert_eq!check.Line 239 uses
assert!(matches!(...))without a preceding log/debug line. Please add a quickdbg!(&outcome)(or equivalent) right before the assertion.As per coding guidelines: "Log before non-
assert_eq!assertions,dbg!complex structures, skip logging for simple scalarassert_eq!."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs` around lines 239 - 242, Add a debug log right before the non-assert_eq! assertion: insert a dbg!(&outcome) (or equivalent logging) immediately before the assert!(matches!(outcome, FetchFullMetadataOutcome::NotModified)) line so the complex `outcome` value is printed when the test fails; locate the assertion using the symbol `outcome` and the pattern `FetchFullMetadataOutcome::NotModified` in the test and place the dbg! call immediately above it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pacquet/crates/package-manager/src/install_with_fresh_lockfile.rs`:
- Around line 471-480: The current drop(resolver); drop(npm_resolver); doesn't
free the resolver's internal Arc caches because local Arcs (meta_cache,
fetch_locker, picked_manifest_cache) remain live; explicitly drop those Arc
bindings before the install pass instead of (or in addition to) dropping the
resolver wrapper. Locate the local variables meta_cache, fetch_locker, and
picked_manifest_cache in install_with_fresh_lockfile.rs and call
drop(meta_cache); drop(fetch_locker); drop(picked_manifest_cache); (or move them
out with std::mem::take/Option and then drop) immediately after dropping
resolver/npm_resolver so the resolver-side packument cache is actually released.
In `@pacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rs`:
- Around line 123-133: Update the comment to accurately reflect the control
flow: instead of saying later visitors “skip the recursion,” state that
concurrent visitors still build per-occurrence tree nodes and run child
resolution, but reuse the shared ResolvedPackage envelope stored in the shared
HashMap protected by the std::sync::Mutex; reference the use of try_join_all for
parallel sibling resolution, the shared HashMap/Mutex dedupe gate, the
ResolvedPackage envelope being reused, and that each visit still allocates a
fresh DependenciesTreeNode and resolves children (i.e., no short-circuit skip of
recursion).
In `@pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs`:
- Line 388: The introduced #[allow(clippy::too_many_arguments)] on
npm_resolver.rs should not be added without justification—either remove it and
refactor the affected function(s) to take a single options/context struct (e.g.,
create a NpmResolveOptions or ResolverContext type and replace the long
parameter list in the target function(s) and their call sites), or keep the
attribute but add an inline comment above it documenting the concrete, temporary
reason for the suppression and a TODO with a ticket/PR reference and target
milestone for the refactor; update the function signature(s) and all callers to
use the new struct if you choose refactoring.
In `@pacquet/crates/resolving-npm-resolver/src/pick_package.rs`:
- Around line 33-51: The doc comment contains Unicode ellipsis characters (“…”)
that Dylint rejects; replace all occurrences of “…” with an ASCII three-dot
sequence ("...") in the module-level comments and any inline docs in this file
(notably the comments describing PackumentFetchLocker,
PickPackageContext::fetch_locker and PackageMetaCache::set behavior and the
other doc-comment blocks around the packument/concurrency explanation),
including the other comment blocks referenced, so all doc comments use "..."
instead of the Unicode ellipsis.
---
Nitpick comments:
In `@pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs`:
- Around line 239-242: Add a debug log right before the non-assert_eq!
assertion: insert a dbg!(&outcome) (or equivalent logging) immediately before
the assert!(matches!(outcome, FetchFullMetadataOutcome::NotModified)) line so
the complex `outcome` value is printed when the test fails; locate the assertion
using the symbol `outcome` and the pattern
`FetchFullMetadataOutcome::NotModified` in the test and place the dbg! call
immediately above it.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 032d75e4-e790-441c-bd49-ebb5cf3f6d7a
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (40)
pacquet/crates/cli/src/cli_args/add.rspacquet/crates/cli/src/cli_args/install.rspacquet/crates/cli/src/state.rspacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rspacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rspacquet/crates/engine-runtime-node-resolver/src/node_resolver.rspacquet/crates/package-manager/src/add.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rspacquet/crates/package-manager/src/install.rspacquet/crates/package-manager/src/install/tests.rspacquet/crates/package-manager/src/install_package_from_registry.rspacquet/crates/package-manager/src/install_package_from_registry/tests.rspacquet/crates/package-manager/src/install_with_fresh_lockfile.rspacquet/crates/resolving-deps-resolver/src/dependencies_graph.rspacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rspacquet/crates/resolving-deps-resolver/src/resolve_importer.rspacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rspacquet/crates/resolving-deps-resolver/src/resolve_peers.rspacquet/crates/resolving-deps-resolver/src/resolved_tree.rspacquet/crates/resolving-deps-resolver/src/tests.rspacquet/crates/resolving-local-resolver/src/local_resolver.rspacquet/crates/resolving-npm-resolver/Cargo.tomlpacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rspacquet/crates/resolving-npm-resolver/src/lib.rspacquet/crates/resolving-npm-resolver/src/mirror.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver.rspacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rspacquet/crates/resolving-npm-resolver/src/pick_package.rspacquet/crates/resolving-npm-resolver/src/pick_package/tests.rspacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rspacquet/crates/resolving-npm-resolver/tests/chain.rspacquet/crates/resolving-resolver-base/src/lib.rspacquet/crates/resolving-resolver-base/src/resolve.rspacquet/crates/tarball/src/lib.rspacquet/crates/tarball/src/tests.rs
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
- GitHub Check: ubuntu-latest / Node.js 24 / Test
- GitHub Check: Lint and Test (windows-latest)
- GitHub Check: Lint and Test (macos-latest)
- GitHub Check: Run benchmark on ubuntu-latest
- GitHub Check: Code Coverage
- GitHub Check: Run benchmark on ubuntu-latest
🧰 Additional context used
📓 Path-based instructions (2)
pacquet/**/*.rs
📄 CodeRabbit inference engine (pacquet/AGENTS.md)
pacquet/**/*.rs: When porting a function that firespnpm:<channel>events throughglobalLogger,logger.debug(), orstreamParser.write(), mirror the call site, payload, and ordering so the reporter parses pacquet's NDJSON the same way it parses pnpm's.
Declare a newtype wrapper for branded string types. Do not collapse the brand into a plainStringor&str.
If upstream always validates before construction, validate in pacquet's wrapper too. The wrapper must construct only viaTryFrom<String>and/orFromStr. Do not provide an infallible public constructor.
If upstream never validates, just brand for type-safety. Expose an infallibleFrom<String>(andFrom<&str>when convenient).
If upstream occasionally constructs without validation, exposefrom_str_uncheckedas an escape hatch alongside the validating constructor.
Match upstream serde behavior for branded types that cross JSON, YAML, or INI boundaries. Use#[serde(try_from = "String")]for deserialization and#[serde(into = "String")]for serialization.
Use#[derive(derive_more::From)]and#[derive(derive_more::Into)]for mechanical conversion impls. Fall back to manualimplonly when conversion needs custom logic.
String-literal unions should becomeenums, not newtype wrappers. Model closed sets of valid string values as enums.
Template literal types should be treated as branded strings with validation discipline from rules 2-5.
Choose owned vs. borrowed parameters to minimize copies. Widen to the most encompassing type (&Pathover&PathBuf,&strover&String) when it doesn't force extra copies.
PreferArc::clone(&x)/Rc::clone(&x)overx.clone()for reference-counted types, so the cost is visible at the call site.
Follow Rust API Guidelines for naming conventions.
Do not use star imports inside module bodies. Writeuse super::{Foo, bar}instead ofuse super::*;. Two forms stay allowed: external-crate preludes likeuse rayon::prelude::*;and root-of-module re-...
Files:
pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rspacquet/crates/resolving-deps-resolver/src/resolved_tree.rspacquet/crates/resolving-deps-resolver/src/tests.rspacquet/crates/resolving-npm-resolver/src/mirror.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rspacquet/crates/resolving-deps-resolver/src/resolve_peers.rspacquet/crates/engine-runtime-node-resolver/src/node_resolver.rspacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rspacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rspacquet/crates/cli/src/cli_args/install.rspacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rspacquet/crates/resolving-npm-resolver/tests/chain.rspacquet/crates/resolving-npm-resolver/src/lib.rspacquet/crates/cli/src/cli_args/add.rspacquet/crates/tarball/src/tests.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rspacquet/crates/resolving-local-resolver/src/local_resolver.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rspacquet/crates/tarball/src/lib.rspacquet/crates/cli/src/state.rspacquet/crates/resolving-resolver-base/src/resolve.rspacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rspacquet/crates/resolving-resolver-base/src/lib.rspacquet/crates/package-manager/src/install_package_from_registry.rspacquet/crates/package-manager/src/add.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rspacquet/crates/package-manager/src/install_package_from_registry/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rspacquet/crates/package-manager/src/install.rspacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rspacquet/crates/resolving-deps-resolver/src/resolve_importer.rspacquet/crates/package-manager/src/install_with_fresh_lockfile.rspacquet/crates/resolving-npm-resolver/src/pick_package/tests.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rspacquet/crates/resolving-npm-resolver/src/pick_package.rspacquet/crates/package-manager/src/install/tests.rs
pacquet/**/{tests,test}/**/*.rs
📄 CodeRabbit inference engine (pacquet/AGENTS.md)
pacquet/**/{tests,test}/**/*.rs: Prefer real fixtures; reach for the dependency-injection seam only when they can't cover the branch. Most happy paths and error paths should be tested withtempfile::TempDir, the mocked registry, or an integration test that spawns the actual binary.
Use the DI seam — a capability trait on theHostprovider, threaded asSys: <Bounds>— only for branches a real fixture can't reach portably: filesystem error kinds, deterministic time, shared process-global state, or external-service happy paths.
Log before non-assert_eq!assertions,dbg!complex structures, skip logging for simple scalarassert_eq!. Follow the test-logging guidance in CODE_STYLE_GUIDE.md.
Tests must not be tolerant of missing build/runtime environment by silently returning early when a tool isn't found. If the test needs a tool, let it panic when the tool is absent.
Prefer#[cfg_attr(target_os = "windows", ignore = "...")](or#[cfg(unix)]) over a runtime probe-and-skip helper for platform-locked tools.
Port relevant tests from pnpm when porting behavior. Matching test coverage is the easiest way to prove behavioral parity.
When porting behavior from pnpm, consult plans/TEST_PORTING.md before adding ported tests. Useallow_known_failure!at the not-yet-implemented boundary and update checkboxes as items land.
Files:
pacquet/crates/resolving-npm-resolver/tests/chain.rs
🧠 Learnings (2)
📚 Learning: 2026-05-20T19:40:55.051Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11774
File: pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs:0-0
Timestamp: 2026-05-20T19:40:55.051Z
Learning: In the pacquet Rust code, ensure the semver implementation uses the `node-semver` crate (not `nodejs-semver`). `node-semver`’s public API does not include a `satisfies_with_prerelease`-style method; prerelease-tolerant matching should be implemented inline by first calling `Range::satisfies`, and when it rejects a prerelease version, retry matching against a stripped `MAJOR.MINOR.PATCH` base of the prerelease version.
Applied to files:
pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rspacquet/crates/resolving-deps-resolver/src/resolved_tree.rspacquet/crates/resolving-deps-resolver/src/tests.rspacquet/crates/resolving-npm-resolver/src/mirror.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rspacquet/crates/resolving-deps-resolver/src/resolve_peers.rspacquet/crates/engine-runtime-node-resolver/src/node_resolver.rspacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rspacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rspacquet/crates/cli/src/cli_args/install.rspacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rspacquet/crates/resolving-npm-resolver/tests/chain.rspacquet/crates/resolving-npm-resolver/src/lib.rspacquet/crates/cli/src/cli_args/add.rspacquet/crates/tarball/src/tests.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rspacquet/crates/resolving-local-resolver/src/local_resolver.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rspacquet/crates/tarball/src/lib.rspacquet/crates/cli/src/state.rspacquet/crates/resolving-resolver-base/src/resolve.rspacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rspacquet/crates/resolving-resolver-base/src/lib.rspacquet/crates/package-manager/src/install_package_from_registry.rspacquet/crates/package-manager/src/add.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rspacquet/crates/package-manager/src/install_package_from_registry/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rspacquet/crates/package-manager/src/install.rspacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rspacquet/crates/resolving-deps-resolver/src/resolve_importer.rspacquet/crates/package-manager/src/install_with_fresh_lockfile.rspacquet/crates/resolving-npm-resolver/src/pick_package/tests.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rspacquet/crates/resolving-npm-resolver/src/pick_package.rspacquet/crates/package-manager/src/install/tests.rs
📚 Learning: 2026-05-20T23:07:58.444Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11784
File: pacquet/crates/resolving-deps-resolver/src/hoist_peers.rs:120-133
Timestamp: 2026-05-20T23:07:58.444Z
Learning: When reviewing code in this pacquet Rust port, follow the upstream pnpm compatibility rule: only match pnpm’s behavior exactly. Do not propose review changes that intentionally deviate from pnpm’s documented/observed behavior, even if pnpm appears buggy. If you identify a real bug in pnpm behavior, the review should prioritize fixing it upstream in pnpm first, and avoid implementing a pnpm-behavior workaround here unless the same fix has already landed upstream.
Applied to files:
pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rspacquet/crates/resolving-deps-resolver/src/resolved_tree.rspacquet/crates/resolving-deps-resolver/src/tests.rspacquet/crates/resolving-npm-resolver/src/mirror.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rspacquet/crates/resolving-deps-resolver/src/resolve_peers.rspacquet/crates/engine-runtime-node-resolver/src/node_resolver.rspacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rspacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rspacquet/crates/cli/src/cli_args/install.rspacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rspacquet/crates/resolving-npm-resolver/tests/chain.rspacquet/crates/resolving-npm-resolver/src/lib.rspacquet/crates/cli/src/cli_args/add.rspacquet/crates/tarball/src/tests.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rspacquet/crates/resolving-local-resolver/src/local_resolver.rspacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rspacquet/crates/tarball/src/lib.rspacquet/crates/cli/src/state.rspacquet/crates/resolving-resolver-base/src/resolve.rspacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rspacquet/crates/resolving-resolver-base/src/lib.rspacquet/crates/package-manager/src/install_package_from_registry.rspacquet/crates/package-manager/src/add.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rspacquet/crates/package-manager/src/install_package_from_registry/tests.rspacquet/crates/resolving-npm-resolver/src/npm_resolver.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rspacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rspacquet/crates/package-manager/src/install.rspacquet/crates/resolving-deps-resolver/src/resolve_dependency_tree.rspacquet/crates/resolving-deps-resolver/src/resolve_importer.rspacquet/crates/package-manager/src/install_with_fresh_lockfile.rspacquet/crates/resolving-npm-resolver/src/pick_package/tests.rspacquet/crates/package-manager/src/dependencies_graph_to_lockfile/tests.rspacquet/crates/resolving-npm-resolver/src/pick_package.rspacquet/crates/package-manager/src/install/tests.rs
🔇 Additional comments (33)
pacquet/crates/resolving-deps-resolver/src/dependencies_graph.rs (1)
26-31: LGTM!pacquet/crates/resolving-npm-resolver/Cargo.toml (1)
41-41: LGTM!pacquet/crates/resolving-deps-resolver/src/resolved_tree.rs (1)
83-89: LGTM!pacquet/crates/resolving-deps-resolver/src/tests.rs (1)
56-56: LGTM!pacquet/crates/resolving-npm-resolver/src/mirror.rs (1)
235-272: LGTM!pacquet/crates/package-manager/src/dependencies_graph_to_lockfile.rs (1)
266-266: LGTM!pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs (1)
34-34: LGTM!Also applies to: 493-493
pacquet/crates/engine-runtime-node-resolver/src/node_resolver.rs (1)
158-158: LGTM!Also applies to: 196-199
pacquet/crates/engine-runtime-bun-resolver/src/bun_resolver.rs (1)
112-112: LGTM!Also applies to: 160-163
pacquet/crates/engine-runtime-deno-resolver/src/deno_resolver.rs (1)
119-119: LGTM!Also applies to: 167-170
pacquet/crates/cli/src/cli_args/install.rs (1)
172-172: LGTM!Also applies to: 229-229
pacquet/crates/resolving-npm-resolver/src/resolve_from_workspace.rs (1)
269-269: LGTM!pacquet/crates/resolving-npm-resolver/tests/chain.rs (1)
25-25: LGTM!Also applies to: 41-42
pacquet/crates/resolving-npm-resolver/src/lib.rs (1)
45-47: LGTM!Also applies to: 62-65
pacquet/crates/cli/src/cli_args/add.rs (1)
93-96: LGTM!Also applies to: 115-115
pacquet/crates/tarball/src/tests.rs (1)
1536-1542: LGTM!Also applies to: 1549-1549, 1636-1642
pacquet/crates/resolving-npm-resolver/src/named_registry_resolver/tests.rs (1)
13-16: LGTM!Also applies to: 74-75
pacquet/crates/resolving-npm-resolver/src/npm_resolver/tests.rs (1)
13-16: LGTM!Also applies to: 66-67
pacquet/crates/resolving-local-resolver/src/local_resolver.rs (1)
79-79: LGTM!Also applies to: 266-266
pacquet/crates/resolving-npm-resolver/src/named_registry_resolver.rs (1)
72-80: LGTM!Also applies to: 158-158, 213-213
pacquet/crates/tarball/src/lib.rs (1)
1782-1782: LGTM!Also applies to: 1795-1801, 1817-1817
pacquet/crates/cli/src/state.rs (1)
10-10: LGTM!Also applies to: 14-18, 75-75
pacquet/crates/resolving-resolver-base/src/resolve.rs (1)
11-11: LGTM!Also applies to: 239-248, 274-277, 325-325
pacquet/crates/resolving-deps-resolver/src/resolve_importer/tests.rs (1)
55-55: LGTM!pacquet/crates/resolving-resolver-base/src/lib.rs (1)
31-34: LGTM!pacquet/crates/package-manager/src/install_package_from_registry.rs (1)
50-57: LGTM!Also applies to: 118-118, 151-151, 180-180, 228-228, 255-255
pacquet/crates/package-manager/src/add.rs (1)
20-20: LGTM!Also applies to: 54-54
pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata_cached.rs (1)
30-31: LGTM!Also applies to: 119-119, 151-153
pacquet/crates/package-manager/src/install_package_from_registry/tests.rs (1)
7-10: LGTM!Also applies to: 103-105, 160-160, 246-246, 268-268, 360-360, 462-462
pacquet/crates/resolving-npm-resolver/src/npm_resolver.rs (1)
96-107: LGTM!Also applies to: 223-232, 270-279, 306-311, 381-385, 397-433
pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata/tests.rs (1)
3-15: LGTM!Also applies to: 72-77, 104-105, 159-165, 194-195, 206-244
pacquet/crates/resolving-npm-resolver/src/fetch_full_metadata.rs (1)
25-26: LGTM!Also applies to: 40-42, 52-64, 66-82, 87-103, 112-143
pacquet/crates/package-manager/src/install/tests.rs (1)
52-52: LGTM!Also applies to: 119-119, 190-190, 254-254, 335-335, 401-401, 487-487, 627-627, 726-726, 845-845, 966-966, 1062-1062, 1166-1166, 1214-1214, 1304-1304, 1396-1396, 1458-1458, 1521-1521, 1610-1610, 1715-1715, 1774-1774, 1860-1860, 1936-1936, 2018-2018, 2218-2218, 2340-2340, 2438-2438, 2542-2542, 2631-2631, 2722-2722, 2812-2812, 2898-2898, 2990-2990, 3080-3080, 3176-3176, 3274-3274, 3358-3358, 3440-3440, 3508-3508, 3572-3572, 3635-3635, 3701-3701, 3783-3783, 3843-3843, 3926-3926, 3975-3975, 4042-4042, 4112-4112, 4178-4178
…by registry
`PickedManifestCache` is shared across the npm (default + JSR) and
named-registry resolvers, but its key was `{name}@{version}` only.
Two registries can serve different artifacts under the same
`name@version` — a public + private collision, or a fork — so a
collision would hand the second resolver the first resolver's
manifest, breaking the downstream dependency graph / peer
extraction / lockfile metadata.
Scope the key by registry (`{registry}\x00{name}@{version}`), matching
the existing `PackageMetaCache` shape, and add a regression test that
points two resolvers at different mock registries (each serving
distinct `dependencies` for `acme@1.0.0`) through one shared cache.
…install bin linking The fresh-lockfile install path was passing `None` for both `snapshots` and `package_manifests` to `LinkVirtualStoreBins`, falling back to the readdir-driven `run_with_readdir` that walks `<virtual_store_dir>` (or `<store_dir>/links` under GVS) and reads each child's `package.json` from disk per slot. The warm-cache prefetch already pulled every snapshot's bundled manifest out of the SQLite store index ([`PrefetchResult::manifests`]), so the disk reads were redundant. Wire the prefetched manifests through: - Destructure `PrefetchResult` so `manifests` survives past the `await` (was being dropped on `let prefetched_cas_paths = prefetch.cas_paths`). - Always build the freshly-resolved lockfile structure (was gated on GVS); both branches now reuse one `built_lockfile`. The build is ~3 ms on the alotta-files fixture and is what we save below anyway. - Project `prefetched_manifests` (keyed by `<integrity>\t<pkg_id>`) into a `PackageManifests` map (`PkgNameVerPeer.without_peer() → Arc<Value>`) by walking the resolved graph and computing the cache key per node — same key shape as `pacquet_store_dir::store_index_key`. - Drive `LinkVirtualStoreBins` with `snapshots: Some(...)` + `package_manifests: &map`; cold-batch packages (cache miss → downloaded fresh in this run) fall through to a per-child disk read just like before. `packages: None` on purpose: the freshly-built lockfile's `packages:` rows carry an incomplete `has_bin` because the resolver's `PackageVersion` deserializer does not include the `bin` field. Trusting the empty-by-omission `has_bin_set` here would filter out every child and skip bin linking entirely. Once `bin` is threaded through `PackageVersion`, the call site can switch to `built_lockfile.packages.as_ref()` to recover the ~95% slot short-circuit the frozen-lockfile path enjoys. Updated snapshots reflect the now-present `<slot>/node_modules/<pkg>/node_modules/.bin/<pkg>` self-shim the lockfile-driven path writes per pnpm's `linkBinsOfDependencies`. The readdir path skipped it via `link_bins_excluding` — that was the divergence the existing `same_global_virtual_store_layout_*` parity tests called out; the fresh path now matches.
…fetched hits `run_with_mem_cache` was delegating to `run_without_mem_cache`, which ran the prefetched-cas-paths check and returned the hit via `(**cas_paths).clone()` — a deep clone of the inner `HashMap<String, PathBuf>`. `run_with_mem_cache` then wrapped the result in `Arc::new(...)`. On a warm install with 1k+ snapshots that is one full per-file map allocation per snapshot, plus an Arc allocation, when the prefetched `Arc<HashMap<...>>` is already on hand. Lift the prefetched-cas-paths check into `run_with_mem_cache` so a hit returns `Arc::clone(prefetched_entry)` straight through and populates `mem_cache` with the same Arc — no per-entry walk, no fresh `Arc::new`. The inner check inside `run_without_mem_cache` stays for the direct callers (binary / zip archive paths in `install_package_by_snapshot`) that still want an owned `HashMap` they can mutate (e.g. to splice in the synthesized `package.json` for runtime archives).
Three small cleanups flagged on #11837: 1. `install_with_fresh_lockfile.rs` — the `drop(resolver); drop(npm_resolver);` block didn't actually free the resolver-side caches because the local `meta_cache`, `fetch_locker`, and `picked_manifest_cache` `Arc` bindings each still held one strong reference of their own. Drop those bindings explicitly too so the cache shrinks before the install pass starts pulling tarballs. 2. `resolve_dependency_tree.rs` — the doc comment on `resolve_dependency_tree` said later visitors "skip the recursion," but the `Some(existing)` branch only AND-folds the `optional` flag and the code then unconditionally allocates a fresh `DependenciesTreeNode` and recurses on the children. Reword the comment to match what the code actually does (only the `ResolvedPackage` envelope is shared). 3. `npm_resolver.rs` — `build_resolve_result` had 9 positional arguments under `#[allow(clippy::too_many_arguments)]`. Drop the suppression and take a `BuildResolveResult` struct instead so the three call sites read as named-field literals and adding a future field doesn't require touching every caller.
* ci(benchmark): run all six integrated-benchmark scenarios Wires `clean-install`, `full-resolution`, `peek`, and `gvs-warm` into `pacquet-integrated-benchmark.yml` so per-PR runs cover the same scenario set the manual `benchmark.yml` workflow already exercises via `benchmarks/bench.sh`. Requested for #11837, where the perf delta affects the resolution-bound scenarios (`firstInstall`, `withWarmCache`, `withWarmModules`, `updatedDependencies`) that the prior two-scenario set did not measure. Each scenario gets its own step with a 10 min hyperfine timeout (same rationale as the existing steps) and writes per-scenario report copies that the summary step concatenates into `SUMMARY.md`. * ci(benchmark): drop peek and gvs-warm scenarios Keep only the two new no-lockfile scenarios (`clean-install`, `full-resolution`) on top of the existing `frozen-lockfile` and `frozen-lockfile-hot-cache`. #11837's perf change is in the fresh-lockfile install path, which only runs when resolution runs — i.e., exactly the no-lockfile scenarios. `peek` mutates an existing lockfile and `gvs-warm` is a frozen-lockfile variant; neither exercises the affected path, and including them only costs per-PR CI wall time. * fix(bench): pin packages: ['.'] in synthesized pnpm-workspace.yaml The integrated-benchmark clones each pacquet revision's source tree into `<bench_dir>/pacquet/`, which on the pnpm/pnpm monorepo includes upstream test fixtures like `workspace/project-manifest-reader/__fixtures__/invalid-package-json/package.json` — intentionally malformed JSON used to exercise pnpm's manifest reader. Without a `packages:` field, both pnpm's `findPackages.ts:28` and pacquet's `crates/workspace/src/projects.rs:128` default to `[".", "**"]`, so the fresh-resolve install path's `find_workspace_projects` walk descends into the cloned source tree and trips on the bad fixture: Error: pacquet_package_manifest::serialization_error × installing dependencies ╰─▶ expected `,` or `}` at line 3 column 3 The walk only runs on the fresh-lockfile branch (`install.rs:628-630`), which is why frozen-lockfile and frozen-lockfile-hot-cache stay green while clean-install and full-resolution fail every time. Pin `packages: ['.']` in the synthesized manifest so enumeration stays at the workspace root. The benchmark's installs are single-project, so this doesn't narrow anything the install actually needed to see. Fixtures supplied via `--fixture-dir` that already declare `packages:` keep their own value. * ci(benchmark): bump no-lockfile scenarios to 20 min Clean-install and full-resolution go through pacquet's fresh-resolve install path, which is currently ~3-5x slower than pnpm on the `alotta-files` fixture (#11832). Hyperfine's default 1 warmup + 10 timed runs across three benchmark targets (pacquet@HEAD, pacquet@main, system pnpm) projects to ~13 min wallclock for these two scenarios, putting the previous 10 min cap right on the edge. Doubling to 20 min keeps the per-step timeout meaningful as a stuck- install detector without losing CI time when the bench is healthy. The frozen-lockfile steps stay at 10 min — they don't traverse the slower fresh-resolve path. * fix(bench): drop --no-frozen-lockfile from full-resolution scenario Pacquet doesn't expose `--no-frozen-lockfile` (only `--frozen-lockfile`, `--prefer-frozen-lockfile`, and `--no-prefer-frozen-lockfile`). Passing it makes clap reject the install: error: unexpected argument '--no-frozen-lockfile' found tip: a similar argument exists: '--frozen-lockfile' The flag was redundant for this scenario anyway: full-resolution starts every iteration with no lockfile on disk (init() skips the lockfile when `lockfile_enabled()` is false; cleanup removes it; `lockfile=false` in the synthesized npmrc/workspace prevents writing one). With no lockfile present the frozen path is unreachable regardless of the flag, so both tools take fresh resolution by definition. Fold full-resolution into clean-install's bare `install` arm. --------- Co-authored-by: Claude <noreply@anthropic.com>
Resolved two conflicts against upstream changes in `perf(pacquet): close the warm-cache resolve gap to pnpm CLI` (#11837) and `fix(installing.deps-resolver): deterministically order cyclic peer suffixes` (#11826): - `package-manager/src/install_with_fresh_lockfile.rs`: main dropped the `enable_global_virtual_store` conditional and now builds the lockfile unconditionally (renamed `layout_lockfile` → `built_lockfile`, added a tracing span). The clippy `if_then_some_else_none` rewrite this branch added to that block is no longer applicable — took main's version. - `resolving-deps-resolver/src/resolve_peers.rs` (two sites): main changed `NodeId` from a `Copy` newtype to an `enum { Counter(u64), Leaf(Arc<str>) }`, so `Some(node_id)` had to become `Some(node_id.clone())`. Combined with this branch's `if_then_some_else_none` rewrite of the sibling `alias` field. The new `node_id.clone()` does not trip `clippy::clone_on_ref_ptr` because `NodeId` is an enum, not an `Arc`/`Rc`/`Weak`. `engine-runtime-node-resolver/src/node_resolver.rs` and `resolving-npm-resolver/src/resolve_from_workspace.rs` auto-merged; the clippy fixes on those lines survived intact. Verified locally: `cargo clippy --locked --workspace --all-targets -- --deny warnings`, `cargo fmt --check`, `taplo format --check`.
Summary
Closes #11832.
On the
alotta-filesbenchmark (1362 nodes, warm cache, GVS on), pacquet was 3-5× behind the TypeScript pnpm CLI whenever resolution ran (firstInstall,withWarmCache,withWarmModules,updatedDependencies). Wall-clock dropped from ~11.83s to ~5.03s on this branch; pnpm sits at ~4.16s, and the remaining gap is concentrated in the resolver's per-nodepick_packagewalk (3.1s of the 5.03s — see #11843 for thepeekManifestFromStorefollow-up that would close it).The branch is a series of small wins rather than one big rewrite. The original
PrefetchingResolver(commit f375c91) was replaced by a batched store-index prefetch (461a4c0) — same throughput, far less plumbing.What's in this PR
Resolve-phase
PackumentFetchLocker(per-cache-keyDashMap<String, Arc<Semaphore>>) so concurrent picks of the same(registry, name)coalesce into one HTTP GET. Mirrors pnpm'srunLimited(pkgMirror, …)inpickPackage.ts. Pacquet was firing N parallel GETs for the same packument per cluster of cross-referencing deps; now it's one.etag/modifiedso the registry can answer304 Not Modifiedon the abbreviated-to-full re-fetch path.spawn_blockinginstead of running on the tokio worker.PickedManifestCacheArc<DashMap<String, Arc<Value>>>so duplicate picks of the samename@versionreuse the already-serialisedArc<Value>instead of re-runningserde_json::to_value.Package,ResolveResult.manifest, andResolveResultitself are now shared viaArcso the tree walk's per-occurrence clones become refcount bumps.std::sync::MutexonTreeCtx(a7c94a9) — the per-package dedupe gate is a shortHashMapinsert with noawaitinside, so a sync mutex is the right tool. Tokio's async mutex was paying per-acquire overhead once per visit per ctx field on the resolve hot path.Install-phase
SELECT … WHERE key IN (…)againstindex.dbat install start, rayon-parallel verify, drops the SQLite mutex before any fs work. Replaces the per-snapshotspawn_blockingfan-out that was serialising onArc<Mutex<StoreIndex>>and queueing in tokio's blocking pool.pnpm:progressper URL (e54208e) —run_with_mem_cachewas emittingfetchedtwice when the in-memory cache hit; mirrors pnpm'spackageRequestershape where the emit fires exactly once.LinkVirtualStoreBinswith the prefetched bundled-manifests map (and built lockfile snapshots), so per-childpackage.jsondisk reads on warm hits are gone. Skips theread_direnumeration too. Updated snapshots reflect the now-present<slot>/node_modules/<pkg>/node_modules/.bin/<pkg>self-shim the lockfile-driven path writes per pnpm'slinkBinsOfDependencies.run_with_mem_cachereturns the prefetchedArc<HashMap>straight through instead of going viarun_without_mem_cache's deep-clone path + redundantArc::new. At 1k+ snapshots that's one per-file map allocation and oneArc::newsaved per snapshot.Correctness
PickedManifestCachekey (57c3094) — the shared cache key was{name}@{version}only; two registries (default + JSR + named-registry) serving different artifacts under the samename@version(private + public collisions, forks) would hand one resolver the other's manifest. Now keyed{registry}\x00{name}@{version}, matchingPackageMetaCache's shape. Regression test included.Diagnostics
pacquet::install::phasetracingevents withelapsed_msforresolve_importer,prefetch_cas_paths,build_fresh_lockfile,virtual_store_layout_new,install_subtree. Made the profiling for this branch tractable and stays in for future work (the same per-phase trace is what motivated perf(pacquet): port peekManifestFromStore fast path to skip the picker on hot-cache resolves #11843).Review cleanup
build_resolve_resultnow takes aBuildResolveResultstruct instead of 9 positional args (#[allow(clippy::too_many_arguments)]gone); resolver-sideArcbindings are dropped explicitly before the install pass so the packument cache actually frees; theresolve_dependency_treedoc comment was wrong about "skipping the recursion" on dedupe hits.Doc + Dylint fixes
Arc.clone()direct, unicode ellipsis in doc comments fixed for Dylint.Scope
MemCachesemantics unchanged. The cache-key bug fix doesn't change the on-disk lockfile; it only prevents an in-memory mix-up that would otherwise produce a wrongResolveResult.manifestfield.Follow-ups
peekManifestFromStorefast path. The store-index row carries the bundledpackage.json(name, version, deps, bin, engines, etc.) but no publish-time, so the fast path is safe only when nopublished_by/minimumReleaseAgepolicy is in effect, no--update, and the wanted lockfile pins a tarball+integrity. With ~95% of nodes short-circuiting, theresolve_importerphase (currently 3.1s on warm cache) drops dramatically — this is the single biggest unimplemented win and the most likely path to parity with pnpm.Benchmark
Wall-clock progression on the
alotta-fileswarm-cache + GVS-on scenario (this branch vsmain):mainThe remaining ~0.87s gap is concentrated in
resolve_importerand is what #11843 targets.Test plan
cargo check --workspace --all-targetscargo clippy --locked --workspace --all-targets -- --deny warningscargo fmt --all -- --checkcargo nextest run -p pacquet-package-manager— 298/298 passcargo nextest run -p pacquet-cli— 94/94 passcargo nextest run -p pacquet-tarball— 44/44 passcargo nextest run -p pacquet-resolving-npm-resolver— 245/245 pass (includes theshared_manifest_cache_does_not_leak_across_registriesregression test for 57c3094)cargo nextest run -p pacquet-resolving-deps-resolver— 28/28 passpnpm.io/benchmarksagainst this branch once it lands onpacquet-perf.Written by an agent (Claude Code, claude-opus-4-7).