feat(package-manager): BuildModules under hoisted node-linker (#438 slice 7)#520
Conversation
…lice 7) Re-enables `BuildModules` for `nodeLinker: hoisted` installs. Slice 6 landed the hoisted install pipeline but skipped the build phase entirely because `BuildModules` walked virtual-store slot directories that don't exist under hoisted. Slice 7 routes the build phase through the slice 4 walker's per-node `dir` so postinstall scripts can run against the on-disk hoisted tree. Changes: - `BuildModules` gains two fields: `pkg_root_by_key: Option<&HashMap<PackageKey, PathBuf>>` overrides the per-snapshot pkg_root lookup with the slice 4 walker's `DependenciesGraphNode::dir` values; `gather_ancestor_bin_paths: bool` switches `extra_bin_paths` to the new `bin_dirs_in_all_parent_dirs` helper, a port of upstream's `binDirsInAllParentDirs` at https://github.com/pnpm/pnpm/blob/94240bc046/building/after-install/src/index.ts#L476-L487. - `pkg_root_for_key` helper: routes between the layout-based slot computation (isolated) and the override map (hoisted). Hoisted snapshots absent from the map (walker-dropped) take the same exit as the isolated `!pkg_dir.exists()` skip. - `InstallFrozenLockfile::run` now builds a snapshot-key → first-recorded-dir map from `walker_result.graph.values()` and threads it (plus `gather_ancestor_bin_paths: true`) into `BuildModules` instead of skipping the phase. Multiple graph nodes with the same dep_path collapse to the first entry, matching upstream's `pkgRoots[0]` pick at https://github.com/pnpm/pnpm/blob/94240bc046/building/after-install/src/index.ts#L348. - New private `HoistedLinkerOutput` struct bundles `hoisted_locations` and `pkg_root_by_key` so the hoisted-branch return doesn't trip `clippy::type_complexity`. Side-effects-cache key shape is unchanged — it's keyed by `pkg_id_with_patch_hash` + dep-graph hash, both layout-independent (`crates/graph-hasher/src/dep_state.rs`). `MISSING_HOISTED_LOCATIONS` is intentionally deferred — pacquet has no `rebuild` command, so the install path always re-runs the walker and never reads `.modules.yaml.hoisted_locations`. Tracked as a follow-up for when `pacquet rebuild` lands. Workspace-aware hoisting (slice 9) and `hoistingLimits` / `externalDependencies` (slice 10) remain. Tests: - Three new helper tests pin `bin_dirs_in_all_parent_dirs` against top-level, conflict-nested, and scoped-package shapes. - Three new helper tests pin `pkg_root_for_key` for the isolated pass-through, the hoisted override hit, and the hoisted-missing short-circuit.
📝 WalkthroughWalkthroughThis PR extends the build-modules system to support hoisted ChangesHoisted Build Module Support
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/package-manager/src/build_modules.rs (1)
852-869: 💤 Low valuePotential infinite loop when
pkg_rootis not underlockfile_dir.If
pkg_rootis not a descendant oflockfile_dir(e.g., due to symlinks or misconfiguration), the loop termination conditiondir == *lockfile_dirwill never be satisfied, and the loop will run untildir.as_os_str().is_empty()after exhausting all ancestors up to the filesystem root. This is likely benign (the loop still terminates), but the resultingbin_dirsvector will contain many irrelevant paths.Consider whether an explicit ancestor check or iteration limit would be appropriate for defensive coding, though this matches upstream's behavior which also walks to root.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/package-manager/src/build_modules.rs` around lines 852 - 869, The loop in bin_dirs_in_all_parent_dirs can walk all the way to filesystem root when pkg_root is not a descendant of lockfile_dir; to fix, first check whether lockfile_dir is an ancestor of pkg_root (e.g., pkg_root.ancestors().any(|a| a == lockfile_dir) or Path::starts_with), and if it is, keep the existing parent-walking loop, but if it is not, avoid the full walk and only emit the local bin dir for pkg_root and the lockfile_dir bin (push pkg_root.join("node_modules").join(".bin") and lockfile_dir.join("node_modules").join(".bin") only); update bin_dirs_in_all_parent_dirs to use that ancestor check (refers to pkg_root, lockfile_dir, and function bin_dirs_in_all_parent_dirs) so we don’t accumulate irrelevant paths when the two paths aren’t related.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/package-manager/src/build_modules.rs`:
- Around line 852-869: The loop in bin_dirs_in_all_parent_dirs can walk all the
way to filesystem root when pkg_root is not a descendant of lockfile_dir; to
fix, first check whether lockfile_dir is an ancestor of pkg_root (e.g.,
pkg_root.ancestors().any(|a| a == lockfile_dir) or Path::starts_with), and if it
is, keep the existing parent-walking loop, but if it is not, avoid the full walk
and only emit the local bin dir for pkg_root and the lockfile_dir bin (push
pkg_root.join("node_modules").join(".bin") and
lockfile_dir.join("node_modules").join(".bin") only); update
bin_dirs_in_all_parent_dirs to use that ancestor check (refers to pkg_root,
lockfile_dir, and function bin_dirs_in_all_parent_dirs) so we don’t accumulate
irrelevant paths when the two paths aren’t related.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 73178cbd-4205-4024-90c5-1d90dc3b72cb
📒 Files selected for processing (3)
crates/package-manager/src/build_modules.rscrates/package-manager/src/build_modules/tests.rscrates/package-manager/src/install_frozen_lockfile.rs
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
- GitHub Check: copilot-pull-request-reviewer
- GitHub Check: Run benchmark on ubuntu-latest
- GitHub Check: Run benchmark on ubuntu-latest
- GitHub Check: Code Coverage
- GitHub Check: Lint and Test (ubuntu-latest)
- GitHub Check: Lint and Test (windows-latest)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs
📄 CodeRabbit inference engine (AGENTS.md)
**/*.rs: Preserve existing method chains andpipe-traitchains; do not break them into intermediateletbindings unless there is a concrete justification such as a compilation failure, borrow checker rejection, meaningful performance improvement, or other technical necessity. Refactoring for style alone is not sufficient justification.
Choose owned vs. borrowed parameters to minimize copies; prefer borrowed types (&Pathover&PathBuf,&strover&String) when it does not force extra copies.
PreferArc::clone(&x)andRc::clone(&x)overx.clone()for reference-counted types to make the cost visible at the call site.
Do not use star imports inside module bodies. Writeuse super::{Foo, bar}instead ofuse super::*;for any glob whose target is a module you control. External-crate preludes (e.g.,use rayon::prelude::*;) and root-of-module re-exports (e.g.,pub use submodule::*;inlib.rs) are exceptions.
Follow Rust API Guidelines for naming, as documented in https://rust-lang.github.io/api-guidelines/naming.html.
Declare a newtype wrapper for any branded string type being ported from TypeScript pnpm. Do not collapse the brand into a plainStringor&str; give the type its own struct so misuse is a type error.
When porting branded string types where upstream TypeScript always validates before construction, validate in the Rust port too. Construct the wrapper only viaTryFrom<String>and/orFromStr; do not provide an infallible public constructor that takes an arbitrary string.
For branded string types where upstream TypeScript never validates (used purely for type-safety to prevent confusion between string slots), expose an infallibleFrom<String>andFrom<&str>constructor in the Rust wrapper.
When upstream TypeScript occasionally constructs a branded type without validation (via bareasassertion), add afrom_str_unchecked(or similarly named) constructor on the Rust side. Keep the validating constructor as well; `from_str_u...
Files:
crates/package-manager/src/install_frozen_lockfile.rscrates/package-manager/src/build_modules.rscrates/package-manager/src/build_modules/tests.rs
🧠 Learnings (5)
📚 Learning: 2026-05-07T23:19:08.272Z
Learnt from: KSXGitHub
Repo: pnpm/pacquet PR: 401
File: tasks/integrated-benchmark/src/work_env.rs:343-344
Timestamp: 2026-05-07T23:19:08.272Z
Learning: When reviewing Rust code in pnpm/pacquet for deprecated API usage, do not automatically treat `serde_saphyr::to_string` as deprecated. In `serde-saphyr` v0.0.25, `serde_saphyr::to_string` has no `#[deprecated]` attribute (the `#[deprecated]` later in `serde-saphyr-0.0.25/src/lib.rs` applies to a different function). Only flag `serde_saphyr::to_string` as deprecated if the resolved dependency version’s source shows `#[deprecated]` on that specific function.
Applied to files:
crates/package-manager/src/install_frozen_lockfile.rscrates/package-manager/src/build_modules.rscrates/package-manager/src/build_modules/tests.rs
📚 Learning: 2026-05-13T22:52:32.579Z
Learnt from: zkochan
Repo: pnpm/pacquet PR: 506
File: crates/package-manager/src/link_bins.rs:238-241
Timestamp: 2026-05-13T22:52:32.579Z
Learning: In Rust code using `matches!` (or `match`) on a borrowed value (e.g., `meta: &PackageMetadata` and a field access like `meta.resolution`), it is safe to use wildcard/OR patterns such as `LockfileResolution::Binary(_) | LockfileResolution::Variations(_)` when the pattern does not bind the inner payload by value. `_` wildcard patterns (and similar discriminant-only patterns) should not move non-`Copy` data; they only test the variant and avoid ownership transfer. Do not flag these patterns as move-out-of-borrow/ownership compile errors. Only patterns that bind the inner value by value (e.g., `LockfileResolution::Binary(b)`) would require ownership and may trigger move errors when matching on data behind a borrow.
Applied to files:
crates/package-manager/src/install_frozen_lockfile.rscrates/package-manager/src/build_modules.rscrates/package-manager/src/build_modules/tests.rs
📚 Learning: 2026-05-13T19:22:48.951Z
Learnt from: zkochan
Repo: pnpm/pacquet PR: 478
File: crates/package-manager/src/hoisted_dep_graph.rs:51-55
Timestamp: 2026-05-13T19:22:48.951Z
Learning: When reviewing this Rust codebase, avoid introducing/using a newtype like `PkgIdWithPatchHash` in only one module (e.g., `hoisted_dep_graph.rs`) if other related pacquet modules still represent the upstream pnpm “pkg id with patch hash” as a plain `String` (e.g., `virtual_store_layout`). For type consistency, either keep `pkg_id_with_patch_hash` as `String` here, or require a workspace-wide sweep that defines/extracts the `PkgIdWithPatchHash` newtype once and updates all call sites together (otherwise defer the refactor to a coordinated follow-up PR).
Applied to files:
crates/package-manager/src/install_frozen_lockfile.rscrates/package-manager/src/build_modules.rscrates/package-manager/src/build_modules/tests.rs
📚 Learning: 2026-05-13T20:09:22.171Z
Learnt from: zkochan
Repo: pnpm/pacquet PR: 486
File: crates/package-manager/src/hoisted_dep_graph.rs:472-476
Timestamp: 2026-05-13T20:09:22.171Z
Learning: In pnpm/pacquet, when generating serialized `hoisted_locations` strings (used for `.modules.yaml`), normalize path separators to forward slashes (e.g., replace `\\` with `/`). This preserves cross-platform portability: the output must not use OS-native separators because upstream pnpm’s `path.relative` can return platform-specific separators, which would break consistency with pnpm/pacquet’s other serialized formats on Windows.
Applied to files:
crates/package-manager/src/install_frozen_lockfile.rscrates/package-manager/src/build_modules.rscrates/package-manager/src/build_modules/tests.rs
📚 Learning: 2026-05-01T10:01:33.766Z
Learnt from: zkochan
Repo: pnpm/pacquet PR: 349
File: crates/reporter/src/tests.rs:121-121
Timestamp: 2026-05-01T10:01:33.766Z
Learning: In Rust test code, follow the repo’s CODE_STYLE_GUIDE test-logging rule: add logging (e.g., `eprintln!`/`eprintln!(...)`) so that useful diagnostic values are printed when a test fails, unless the assertion is `assert_eq!` (where the differing values are already included). Concretely, if you use assertions like `assert!`, `assert_ne!`, etc., ensure the test logs the relevant actual/expected values (or context) before/around the assertion so failures can be diagnosed without rerunning.
Applied to files:
crates/package-manager/src/build_modules/tests.rs
🔇 Additional comments (11)
crates/package-manager/src/build_modules.rs (4)
263-287: LGTM!
317-318: LGTM!Also applies to: 341-352, 460-461
636-654: LGTM!
807-831: LGTM!crates/package-manager/src/build_modules/tests.rs (3)
313-314: LGTM!Also applies to: 382-383, 439-440, 520-521, 652-653, 716-717, 773-774, 1007-1008, 1116-1117, 1234-1235, 1462-1463, 1568-1569, 1645-1646
1656-1729: LGTM!
1731-1800: LGTM!crates/package-manager/src/install_frozen_lockfile.rs (4)
651-663: LGTM!
734-755: LGTM!
999-1035: LGTM!
1098-1115: LGTM!
Micro-Benchmark ResultsLinux |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #520 +/- ##
=======================================
Coverage 88.51% 88.52%
=======================================
Files 125 125
Lines 13821 13864 +43
=======================================
+ Hits 12234 12273 +39
- Misses 1587 1591 +4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Integrated-Benchmark Report (Linux)Scenario: Frozen Lockfile
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 2.5021774692000003,
"stddev": 0.08620260968769866,
"median": 2.5148198645999997,
"user": 2.6243643999999997,
"system": 3.5178071399999995,
"min": 2.3575712315999997,
"max": 2.6345944915999997,
"times": [
2.6087735255999998,
2.4029113266,
2.5189491326,
2.6345944915999997,
2.4465173956,
2.5288174345999996,
2.3575712315999997,
2.5453197515999997,
2.4676298055999997,
2.5106905966
]
},
{
"command": "pacquet@main",
"mean": 2.4925537328000003,
"stddev": 0.06997546707701241,
"median": 2.4666305825999997,
"user": 2.6045027000000003,
"system": 3.5013616400000003,
"min": 2.4117120175999998,
"max": 2.6289714795999997,
"times": [
2.5650826556,
2.4117120175999998,
2.6289714795999997,
2.4592015085999996,
2.4712601835999997,
2.4418895536,
2.4839086206,
2.4359937395999998,
2.5655165876,
2.4620009815999997
]
},
{
"command": "pnpm",
"mean": 5.773446385799999,
"stddev": 0.09030373752335401,
"median": 5.7399503706,
"user": 8.4099835,
"system": 4.262454839999999,
"min": 5.6987428426,
"max": 5.9693758926,
"times": [
5.7636001295999995,
5.6987428426,
5.8490111906,
5.7079123125999995,
5.7425791206,
5.7026375756,
5.7373216205999995,
5.9693758926,
5.8588302966,
5.7044528765999996
]
}
]
}Scenario: Frozen Lockfile (Hot Cache)
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.7540791689999999,
"stddev": 0.031881823528990244,
"median": 0.7512794838,
"user": 0.37382321999999996,
"system": 1.6256188999999999,
"min": 0.7180808598,
"max": 0.8337625508,
"times": [
0.8337625508,
0.7512334398,
0.7369499808,
0.7322534868,
0.7180808598,
0.7323919118,
0.7563439028,
0.7684643378,
0.7599856918,
0.7513255278
]
},
{
"command": "pacquet@main",
"mean": 0.8175035339000001,
"stddev": 0.02656020097432925,
"median": 0.8179252283,
"user": 0.35846582,
"system": 1.6560553000000002,
"min": 0.7775215878,
"max": 0.8469090708,
"times": [
0.8460013438,
0.7775215878,
0.8431543848,
0.8407422148,
0.8072789118,
0.7912790718,
0.7896245808,
0.8469090708,
0.8039526278,
0.8285715448
]
},
{
"command": "pnpm",
"mean": 2.4170942548,
"stddev": 0.06868683805605208,
"median": 2.4257619168,
"user": 2.84659962,
"system": 2.1711403,
"min": 2.3215408558,
"max": 2.5441139128,
"times": [
2.5441139128,
2.4602449267999997,
2.3381111908,
2.4337186238,
2.3215408558,
2.4193623818,
2.3798875157999997,
2.3610079028,
2.4807937858,
2.4321614518
]
}
]
} |
Summary
Re-enables
BuildModulesundernodeLinker: hoisted. Slice 6 (#518) wired the hoisted install pipeline but skipped the build phase entirely becauseBuildModuleswalked virtual-store slot directories that don't exist under hoisted. Slice 7 routes the build phase through the slice 4 walker's per-nodedirso postinstall scripts can run against the on-disk hoisted tree, withextra_bin_pathsgathered from every ancestornode_modules/.binup to the install root (npm-style ancestor resolution).Changes
BuildModulesgains two fields:pkg_root_by_key: Option<&HashMap<PackageKey, PathBuf>>— overrides the per-snapshotpkgRootlookup. Populated by the hoisted linker fromDependenciesGraphNode::dir.Nonefor isolated (theVirtualStoreLayout::slot_dirpath is used).gather_ancestor_bin_paths: bool— switchesextra_bin_pathsto the newbin_dirs_in_all_parent_dirshelper, a port of upstream'sbinDirsInAllParentDirs.pkg_root_for_keyhelper routes between layout-based slot computation (isolated) and the override map (hoisted). Hoisted snapshots absent from the map (walker-dropped — pre-skipped, optional skip) take the same exit as the isolated!pkg_dir.exists()skip.InstallFrozenLockfile::runnow builds a snapshot-key → first-recorded-dir map fromwalker_result.graph.values()and threads it (plusgather_ancestor_bin_paths: true) intoBuildModulesinstead of skipping the phase. Multiple graph nodes with the samedep_path(a single physical package nested under siblings via version conflict) collapse to the first entry, matching upstream'spkgRoots[0]pick.HoistedLinkerOutputprivate struct bundleshoisted_locationsandpkg_root_by_keyso the hoisted-branch return doesn't tripclippy::type_complexity.Out of scope
MISSING_HOISTED_LOCATIONS— upstream's error fires when rebuild reads.modules.yaml.hoistedLocationsand finds it absent. Pacquet has norebuildcommand yet, so the install path always re-runs the walker and never consults the persisted field. Tracked as a follow-up for whenpacquet rebuildlands.hoistingLimits/externalDependencies(slice 10) remain.Side-effects-cache key shape is unchanged — it's keyed by
pkg_id_with_patch_hash+ dep-graph hash, both layout-independent (crates/graph-hasher/src/dep_state.rs).Test plan
just ready(1219 tests pass; lint, fmt, typos clean)just dylint(perfectionist) cleantaplo format --checkcleanRUSTDOCFLAGS=\"-D warnings\" cargo doc -p pacquet-package-manager --no-depscleancrates/package-manager/src/build_modules/tests.rs:bin_dirs_top_level_hoisted_pkg,bin_dirs_nested_hoisted_pkg,bin_dirs_scoped_pkg_pushes_every_step— pin the wire-format output of the upstream-mirroring helper.pkg_root_for_key_isolated_uses_layout,pkg_root_for_key_hoisted_uses_override,pkg_root_for_key_hoisted_missing_returns_none— cover the isolated pass-through, the hoisted override hit, and the missing-snapshot short-circuit.End-to-end coverage against the registry-mock with a real lifecycle-script package is left to a follow-up — same status as the slice 6 PR. The slice 6 wiring tests already prove the install-pipeline composition; the slice 7 helper tests prove the per-snapshot lookup contract.
Written by an agent (Claude Code, claude-opus-4-7).
Summary by CodeRabbit
Bug Fixes
Tests