Skip to content

feat(pacquet): port directory fetcher for injected workspace deps#11678

Merged
zkochan merged 3 commits into
mainfrom
directory-fetcher-port
May 16, 2026
Merged

feat(pacquet): port directory fetcher for injected workspace deps#11678
zkochan merged 3 commits into
mainfrom
directory-fetcher-port

Conversation

@zkochan

@zkochan zkochan commented May 16, 2026

Copy link
Copy Markdown
Member

Summary

  • Ports pnpm's fetching/directory-fetcher as a new pacquet-directory-fetcher crate (file walker, manifest read, requires_build detection, broken-symlink ENOENT skip, resolveSymlinks toggle).
  • Wires LockfileResolution::Directory end-to-end through InstallPackageBySnapshot::run and create_virtual_store.rs, so injected workspace deps (file:./local-pkg + dependenciesMeta[*].injected = true) actually install instead of returning UnsupportedResolution { resolution_kind: "directory" }.
  • No CAFS write for directory deps — matches upstream's local: true, packageImportMethod: 'hardlink' semantics: the fetcher returns source-path entries and import_indexed_dir / link_file hardlink-or-copy straight from the source.

What's in scope

  • pacquet-directory-fetcher crate:
    • walk_all_files ↔ upstream's fetchAllFilesFromDir (recursive, node_modules exclusion at any depth, fs::metadata vs symlink_metadata + canonicalize fork, broken-symlink ENOENT skip).
    • walk_package_files ↔ upstream's fetchPackageFilesFromDir (delegates to pacquet_git_fetcher::packlist, tolerates missing manifest for the Bit-workspace shape).
    • DirectoryFetcher::run returns files_map / manifest / requires_build.
  • install_package_by_snapshot.rs: new workspace_root: &Path field threaded down from InstallFrozenLockfile; directory arm resolves dir_resolution.directory against it, calls the fetcher, and returns its files_map as cas_paths.
  • create_virtual_store.rs::snapshot_cache_key: returns Ok(None) for directory resolutions (no warm-cache key — every install re-walks the source).
  • 14 unit + integration tests covering the walker and the public entry point.

What's deferred to follow-ups

  • resolveSymlinksInInjectedDirs / includeOnlyPackageFiles config knobs — currently hard-coded to upstream's defaults (false / false) from extendInstallOptions.ts:41.
  • Injected-dep re-mirror pass — hoisted_dep_graph.rs already collects injection_targets_by_dep_path for every directory-typed snapshot location, but the post-install mirror that rebuilds those copies still needs implementing.
  • package.json5 / package.yaml manifest read — safe_read_package_json_from_dir only handles package.json. Same parity gap as every other manifest read in pacquet; not specific to this PR.

Test plan

  • cargo nextest run -p pacquet-directory-fetcher (14 tests) — passes locally.
  • cargo nextest run (1277 tests) — passes locally.
  • cargo clippy --locked -- --deny warnings — clean.
  • cargo check --locked — clean.
  • Manual smoke install with a workspace containing a file:./local-pkg injected dep (suggested validation; not covered by unit tests).

Written by an agent (Claude Code, claude-opus-4-7).

Summary by CodeRabbit

  • New Features

    • Full support for local directory workspace dependencies: projects can reference and install packages from workspace directories.
    • Choice to fetch either package-listed files or all files for directory dependencies.
    • Automatic detection when a local dependency requires a build; more reliable handling of symlinks and excluded folders (e.g., node_modules).
  • Tests

    • Added unit and integration tests for directory walking, packlist filtering, symlink behaviors, and fetcher outputs.

Review Change Stack

…ry fetcher for injected workspace deps

Port pnpm's `fetching/directory-fetcher` package as a new
`pacquet-directory-fetcher` crate and wire `LockfileResolution::Directory`
into `InstallPackageBySnapshot::run` so injected workspace deps
(`file:./local-pkg` + `dependenciesMeta[*].injected = true`) actually
install instead of returning `UnsupportedResolution`.

Mirrors upstream's
[`fetching/directory-fetcher/src/index.ts`](https://github.com/pnpm/pnpm/blob/85ceff2383/fetching/directory-fetcher/src/index.ts):

- `walk_all_files` ports `fetchAllFilesFromDir` — recursive walk,
  `node_modules` exclusion at any depth, broken-symlink ENOENT skip,
  `resolve_symlinks` toggle between `fileStat` (`fs::metadata`) and
  `realFileStat` (`symlink_metadata` + `canonicalize`).
- `walk_package_files` ports `fetchPackageFilesFromDir` — delegates
  to `pacquet_git_fetcher::packlist` for the npm-packlist filter,
  tolerates a missing manifest for the Bit-workspace shape upstream
  documents at L63-L66.
- `DirectoryFetcher::run` returns `files_map` / `manifest` /
  `requires_build`, matching upstream's `FetchResult` minus
  `local: true` (implicit at the call site) and `packageImportMethod`
  (encoded by which slot the install dispatcher writes to).

Install dispatch:

- `install_package_by_snapshot.rs` resolves `dir_resolution.directory`
  against the newly threaded `workspace_root` (upstream's
  `lockfileDir`), calls `DirectoryFetcher::run`, and hands the
  source-path `files_map` straight through as `cas_paths`. The
  existing `import_indexed_dir` / `link_file` path accepts arbitrary
  source paths so no CAFS write is needed — directory deps don't go
  through the store at all, matching upstream's
  `local: true, packageImportMethod: 'hardlink'` semantics.
- `create_virtual_store.rs`'s `snapshot_cache_key` now returns
  `Ok(None)` for directory resolutions: there's no warm-cache key
  to recover from (the source dir may have changed since last
  install), so every install re-walks. Matches upstream.

Deferred follow-ups (out of scope for this PR):

- `resolveSymlinksInInjectedDirs` / `includeOnlyPackageFiles`
  config-knob plumbing — the dispatch hard-codes the upstream
  defaults (`false` / `false`) from
  [`extendInstallOptions.ts:41`](https://github.com/pnpm/pnpm/blob/85ceff2383/installing/deps-installer/src/install/extendInstallOptions.ts#L41)
  for now.
- Injected-dep re-mirror pass — `hoisted_dep_graph.rs` already
  records `injection_targets_by_dep_path` for every directory-typed
  snapshot location, but the post-install mirror that rebuilds those
  copies is not implemented yet.
- `package.json5` / `package.yaml` manifest read — pacquet's
  `safe_read_package_json_from_dir` only handles `package.json`; the
  other two variants upstream's `safeReadProjectManifestOnly`
  supports are a parity gap that affects directory deps the same way
  it affects every other manifest read in pacquet.

---
Written by an agent (Claude Code, claude-opus-4-7).
Copilot AI review requested due to automatic review settings May 16, 2026 00:54
@coderabbitai

coderabbitai Bot commented May 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6ae878cc-7331-43f6-8207-690f79feadc2

📥 Commits

Reviewing files that changed from the base of the PR and between f2db87c and 99d084c.

📒 Files selected for processing (1)
  • pacquet/crates/package-manager/src/create_virtual_store.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • pacquet/crates/package-manager/src/create_virtual_store.rs
📜 Recent review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
  • GitHub Check: Agent
  • GitHub Check: Lint and Test (ubuntu-latest)
  • GitHub Check: Lint and Test (windows-latest)
  • GitHub Check: Lint and Test (macos-latest)
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Code Coverage
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Compile & Lint

📝 Walkthrough

Walkthrough

Adds a new pacquet-directory-fetcher crate (walker + fetcher), unit and integration tests, and integrates it into package-manager by threading workspace_root and enabling LockfileResolution::Directory via the cold install path.

Changes

Directory-Fetcher Feature

Layer / File(s) Summary
Crate setup and error types
Cargo.toml, pacquet/crates/directory-fetcher/Cargo.toml, pacquet/crates/directory-fetcher/src/error.rs, pacquet/crates/directory-fetcher/src/lib.rs
New workspace crate manifest, library entry, and DirectoryFetcherError with I/O, packlist, and manifest variants; re-exports public API.
Directory walking implementation
pacquet/crates/directory-fetcher/src/walker.rs
FilesMap type and two walkers: walk_all_files (recursive, skip node_modules, optional symlink resolution, symlink-cycle guard, broken-symlink skipping) and walk_package_files (reads package.json, calls pacquet_git_fetcher::packlist).
Walker unit tests
pacquet/crates/directory-fetcher/src/walker/tests.rs
Tests for recursive discovery, node_modules exclusion, empty dirs, symlink cycles, broken symlink behavior (resolve/non-resolve), realpath vs symlink source, and packlist filtering with/without package.json.
Fetcher public API and implementation
pacquet/crates/directory-fetcher/src/fetcher.rs
DirectoryFetcher and DirectoryFetchOutput structs; run() chooses walker mode, reads manifest safely, computes requires_build from scripts and files (e.g., binding.gyp), and returns aggregated output or error.
Fetcher integration tests
pacquet/crates/directory-fetcher/tests/fetcher.rs
End-to-end tests validating all-files and package-files modes, manifest handling, and requires_build detection via install script or binding.gyp.
Package-manager integration and workspace-root threading
pacquet/crates/package-manager/Cargo.toml, pacquet/crates/package-manager/src/create_virtual_store.rs, .../src/install_frozen_lockfile.rs, .../src/install_package_by_snapshot.rs
Adds pacquet-directory-fetcher dependency; threads workspace_root: &Path through CreateVirtualStore and InstallPackageBySnapshot; treats LockfileResolution::Directory as cold-path (Ok(None) cache key) and invokes DirectoryFetcher with a new DirectoryFetch error variant.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

Poem

🐰 I hopped through folders, skipping node_modules tight,
followed soft symlinks by lantern light,
read package.json and packed files just right,
threaded workspace roots to fetch paths bright,
now workspace deps install — hop, fetch, delight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main objective: porting directory fetcher for workspace dependencies in the pacquet package manager.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch directory-fetcher-port

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
pacquet/crates/package-manager/src/create_virtual_store.rs (2)

374-385: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Do not current-lockfile-skip directory snapshots.

Directory resolutions currently pass the skip gate when deps are unchanged and slot exists, because integrity compares as None == None. That prevents re-walking mutable workspace sources on subsequent installs.

Suggested fix
                 let current_metadata =
                     current_packages.and_then(|p| p.get(&snapshot_key.without_peer()));
                 let wanted_metadata = packages.get(&snapshot_key.without_peer());
+                if matches!(
+                    wanted_metadata.map(|m| &m.resolution),
+                    Some(LockfileResolution::Directory(_))
+                ) {
+                    // Directory-backed snapshots are mutable local sources.
+                    // Always re-fetch/re-link.
+                    return true;
+                }
                 if !integrity_equal(current_metadata, wanted_metadata) {
                     return true;
                 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pacquet/crates/package-manager/src/create_virtual_store.rs` around lines 374
- 385, The current skip logic treats None==None as unchanged (integrity_equal
returns true) which lets directory resolutions be skipped; change the check
after computing current_metadata and wanted_metadata so that if
integrity_equal(...) is true you still force a rebuild when the snapshot
represents a directory resolution—detect this via snapshot_key (e.g.,
snapshot_key.is_directory() or by inspecting wanted_metadata/current_metadata
resolution type) and return true for directory snapshots instead of skipping;
update the block around current_metadata, wanted_metadata and the
integrity_equal(...) call (and not the later layout.slot_dir(...) check) to
ensure directory-based snapshots are always considered changed.

1041-1046: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Classify DirectoryFetch as fetch-side for optional snapshots.

Optional snapshot failure swallowing excludes InstallPackageBySnapshotError::DirectoryFetch, so optional injected-directory failures currently hard-fail installs instead of being dropped like other fetch-side failures.

Suggested fix
 fn is_fetch_side_failure(err: &InstallPackageBySnapshotError) -> bool {
     matches!(
         err,
         InstallPackageBySnapshotError::DownloadTarball(_)
-            | InstallPackageBySnapshotError::GitFetch(_),
+            | InstallPackageBySnapshotError::GitFetch(_)
+            | InstallPackageBySnapshotError::DirectoryFetch(_),
     )
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pacquet/crates/package-manager/src/create_virtual_store.rs` around lines 1041
- 1046, The is_fetch_side_failure function currently only treats DownloadTarball
and GitFetch variants as fetch-side; add
InstallPackageBySnapshotError::DirectoryFetch to the matches so optional
snapshot failures from injected directories are classified as fetch-side and can
be dropped like the others; update the matches pattern inside
is_fetch_side_failure to include DirectoryFetch alongside DownloadTarball and
GitFetch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pacquet/crates/directory-fetcher/src/walker.rs`:
- Around line 37-43: The walker currently follows directory symlinks without
tracking visited directories, causing infinite recursion on symlink cycles;
modify walk_all_files and the internal walker (walk_all_inner / any helper that
recurses into directories) to accept and maintain a visited-set of canonical
directory keys (use std::fs::canonicalize(or equivalent) when resolve_symlinks
is true) and before recursing into a directory compute its canonical path, check
the visited set and skip recursion if already present, otherwise insert it and
continue; ensure the visited-set is threaded through all recursive calls so the
same guard is applied where the code currently recurses into directories
(including the other similar walker entry points mentioned).

---

Outside diff comments:
In `@pacquet/crates/package-manager/src/create_virtual_store.rs`:
- Around line 374-385: The current skip logic treats None==None as unchanged
(integrity_equal returns true) which lets directory resolutions be skipped;
change the check after computing current_metadata and wanted_metadata so that if
integrity_equal(...) is true you still force a rebuild when the snapshot
represents a directory resolution—detect this via snapshot_key (e.g.,
snapshot_key.is_directory() or by inspecting wanted_metadata/current_metadata
resolution type) and return true for directory snapshots instead of skipping;
update the block around current_metadata, wanted_metadata and the
integrity_equal(...) call (and not the later layout.slot_dir(...) check) to
ensure directory-based snapshots are always considered changed.
- Around line 1041-1046: The is_fetch_side_failure function currently only
treats DownloadTarball and GitFetch variants as fetch-side; add
InstallPackageBySnapshotError::DirectoryFetch to the matches so optional
snapshot failures from injected directories are classified as fetch-side and can
be dropped like the others; update the matches pattern inside
is_fetch_side_failure to include DirectoryFetch alongside DownloadTarball and
GitFetch.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ac41f0c9-ed91-4ce7-8eea-048fa56605b6

📥 Commits

Reviewing files that changed from the base of the PR and between 496e655 and 31ca1fd.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (12)
  • Cargo.toml
  • pacquet/crates/directory-fetcher/Cargo.toml
  • pacquet/crates/directory-fetcher/src/error.rs
  • pacquet/crates/directory-fetcher/src/fetcher.rs
  • pacquet/crates/directory-fetcher/src/lib.rs
  • pacquet/crates/directory-fetcher/src/walker.rs
  • pacquet/crates/directory-fetcher/src/walker/tests.rs
  • pacquet/crates/directory-fetcher/tests/fetcher.rs
  • pacquet/crates/package-manager/Cargo.toml
  • pacquet/crates/package-manager/src/create_virtual_store.rs
  • pacquet/crates/package-manager/src/install_frozen_lockfile.rs
  • pacquet/crates/package-manager/src/install_package_by_snapshot.rs
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
  • GitHub Check: Agent
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Lint and Test (macos-latest)
  • GitHub Check: Lint and Test (ubuntu-latest)
  • GitHub Check: Lint and Test (windows-latest)
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Code Coverage
  • GitHub Check: Compile & Lint
🧰 Additional context used
📓 Path-based instructions (2)
pacquet/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

No star imports inside module bodies — write use super::{Foo, bar} instead of use super::*; and the same for any other glob. Exception: external-crate preludes like use rayon::prelude::*; and root-of-module re-exports like pub use submodule::*; in lib.rs

Files:

  • pacquet/crates/directory-fetcher/src/lib.rs
  • pacquet/crates/directory-fetcher/src/fetcher.rs
  • pacquet/crates/directory-fetcher/src/walker/tests.rs
  • pacquet/crates/directory-fetcher/tests/fetcher.rs
  • pacquet/crates/package-manager/src/install_package_by_snapshot.rs
  • pacquet/crates/package-manager/src/create_virtual_store.rs
  • pacquet/crates/directory-fetcher/src/error.rs
  • pacquet/crates/directory-fetcher/src/walker.rs
  • pacquet/crates/package-manager/src/install_frozen_lockfile.rs
pacquet/crates/*/tests/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

Tests live alongside the code they exercise (standard Cargo layout) plus integration tests under each crate's tests/ — shared test fixtures live under crates/testing-utils/src/fixtures/

Files:

  • pacquet/crates/directory-fetcher/tests/fetcher.rs
🔇 Additional comments (5)
Cargo.toml (1)

30-30: LGTM!

pacquet/crates/directory-fetcher/Cargo.toml (1)

1-29: LGTM!

pacquet/crates/directory-fetcher/src/error.rs (1)

1-32: LGTM!

pacquet/crates/directory-fetcher/src/lib.rs (1)

1-15: LGTM!

pacquet/crates/directory-fetcher/src/walker/tests.rs (1)

1-206: LGTM!

Comment thread pacquet/crates/directory-fetcher/src/walker.rs
@codecov-commenter

codecov-commenter commented May 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.57252% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.10%. Comparing base (f05845a) to head (99d084c).
⚠️ Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
pacquet/crates/directory-fetcher/src/walker.rs 80.39% 20 Missing ⚠️
...package-manager/src/install_package_by_snapshot.rs 9.09% 10 Missing ⚠️
...crates/package-manager/src/create_virtual_store.rs 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #11678      +/-   ##
==========================================
- Coverage   89.12%   89.10%   -0.03%     
==========================================
  Files         127      129       +2     
  Lines       14470    14590     +120     
==========================================
+ Hits        12897    13000     +103     
- Misses       1573     1590      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Micro-Benchmark Results

Linux

group                          main                                   pr
-----                          ----                                   --
tarball/download_dependency    1.02      7.5±0.31ms   575.4 KB/sec    1.00      7.4±0.56ms   585.1 KB/sec

@github-actions

github-actions Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Integrated-Benchmark Report (Linux)

Scenario: Frozen Lockfile

Command Mean [s] Min [s] Max [s] Relative
pacquet@HEAD 2.405 ± 0.086 2.279 2.555 1.00
pacquet@main 2.432 ± 0.101 2.354 2.671 1.01 ± 0.06
pnpm 4.630 ± 0.111 4.514 4.858 1.93 ± 0.08
BENCHMARK_REPORT.json
{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 2.40500251908,
      "stddev": 0.08587046502617665,
      "median": 2.38828746248,
      "user": 2.7132019,
      "system": 3.6234844399999995,
      "min": 2.27871210848,
      "max": 2.55540654548,
      "times": [
        2.55540654548,
        2.43185934848,
        2.34682322448,
        2.5191348004800003,
        2.36597901148,
        2.39115814748,
        2.45088856348,
        2.38541677748,
        2.3246466634800003,
        2.27871210848
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 2.43193101088,
      "stddev": 0.10099066768577186,
      "median": 2.39287686698,
      "user": 2.6945631999999997,
      "system": 3.60440924,
      "min": 2.3536231834800003,
      "max": 2.67102731248,
      "times": [
        2.35831062248,
        2.38369831448,
        2.52483841048,
        2.67102731248,
        2.48193863248,
        2.37657023248,
        2.4058666574800003,
        2.3536231834800003,
        2.4020554194800003,
        2.3613813234800003
      ]
    },
    {
      "command": "pnpm",
      "mean": 4.63032025238,
      "stddev": 0.11147027941919106,
      "median": 4.60156582598,
      "user": 7.7986065,
      "system": 4.01984844,
      "min": 4.51412682048,
      "max": 4.85831962548,
      "times": [
        4.6213888144799995,
        4.56167876148,
        4.59522942948,
        4.51412682048,
        4.56430111148,
        4.60926252448,
        4.56522647948,
        4.80576673448,
        4.85831962548,
        4.60790222248
      ]
    }
  ]
}

Scenario: Frozen Lockfile (Hot Cache)

Command Mean [ms] Min [ms] Max [ms] Relative
pacquet@HEAD 748.5 ± 48.3 714.1 870.1 1.00
pacquet@main 772.8 ± 51.2 708.4 899.6 1.03 ± 0.10
pnpm 2482.6 ± 103.4 2299.5 2674.7 3.32 ± 0.25
BENCHMARK_REPORT.json
{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 0.7484913258800001,
      "stddev": 0.04830459949056074,
      "median": 0.7333133365800001,
      "user": 0.38173436,
      "system": 1.6393193400000001,
      "min": 0.7141284535800001,
      "max": 0.8700861185800001,
      "times": [
        0.8700861185800001,
        0.7212942975800001,
        0.7272560825800001,
        0.7141284535800001,
        0.7366529175800001,
        0.7335235615800001,
        0.7331031115800001,
        0.7943560245800001,
        0.7381397295800001,
        0.7163729615800001
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 0.7727917494800001,
      "stddev": 0.051224003343681844,
      "median": 0.76804807708,
      "user": 0.37581406,
      "system": 1.63994844,
      "min": 0.7084302385800001,
      "max": 0.8996088265800001,
      "times": [
        0.77058861458,
        0.76915891558,
        0.79991348358,
        0.7368073725800001,
        0.76524094358,
        0.7374460525800001,
        0.8996088265800001,
        0.77378580858,
        0.7669372385800001,
        0.7084302385800001
      ]
    },
    {
      "command": "pnpm",
      "mean": 2.4826345729799995,
      "stddev": 0.10344117310843659,
      "median": 2.49332283558,
      "user": 2.97188316,
      "system": 2.18996014,
      "min": 2.29948185758,
      "max": 2.67472127258,
      "times": [
        2.52341231758,
        2.53147649058,
        2.46120285058,
        2.67472127258,
        2.37484600258,
        2.42405754358,
        2.29948185758,
        2.54033969758,
        2.5335743435799998,
        2.4632333535799997
      ]
    }
  ]
}

…abbit review on #11678

Three parity / hardening fixes flagged in the CodeRabbit review on PR
#11678:

1. **Carve out directory snapshots from the current-lockfile-skip
   gate** (`create_virtual_store.rs`). Directory-typed snapshots have
   `integrity() == None`; without the carve-out `integrity_equal`
   collapsed `None == None == true` and the skip filter dropped the
   snapshot whenever a slot for it existed on disk, so a second
   install never re-walked the (mutable) source dir. Mirrors pnpm's
   `!isDirectoryDep` clause in `depIsPresent` at
   <https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L226-L228>.

2. **Add `DirectoryFetch` to `is_fetch_side_failure`**
   (`create_virtual_store.rs`). Upstream's catch at
   [`lockfileToDepGraph.ts:286-298`](https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L286-L298)
   wraps the whole `fetchPackage` dispatch, so directory-fetcher
   errors on optional snapshots are swallowed uniformly with tarball
   / git fetch errors. Without this, an optional injected-directory
   dep whose source was missing would hard-fail the install instead
   of being dropped.

3. **Symlink-cycle guard in `walk_all_inner`** (`directory-fetcher/
   src/walker.rs`). A `loop -> .` (or any ancestor-pointing) symlink
   previously sank the walker into infinite recursion until either
   ENAMETOOLONG or stack overflow fired. Skip-on-revisit keyed off
   `fs::canonicalize`, matching the pattern
   `pacquet_git_fetcher::packlist` already uses for
   `bundleDependencies` cycles. Pnpm's directory-fetcher has the same
   vulnerability; the guard is a defensible divergence because the
   positive-case behavior is identical to pnpm and the cycle case
   degrades from "crash" to "skip with a `tracing::warn`".

Added a regression test
(`walk_all_files_terminates_on_symlink_cycle`) that points a
`loop -> root` symlink at the walk root and asserts the cycle guard
short-circuits before any `loop/` descendant is recorded.

---
Written by an agent (Claude Code, claude-opus-4-7).
@zkochan

zkochan commented May 16, 2026

Copy link
Copy Markdown
Member Author

Addressed all three CodeRabbit findings in f2db87c:

  1. Walker symlink-cycle guard (pacquet/crates/directory-fetcher/src/walker.rs) — added a visited-canonical-path HashSet to walk_all_inner, matching the pattern pacquet_git_fetcher::packlist already uses for bundleDependencies cycles. New regression test walk_all_files_terminates_on_symlink_cycle covers the loop -> root case.

  2. Directory carve-out in current-lockfile-skip (create_virtual_store.rs:366-394) — directory snapshots now bypass the skip gate (forced through the cold path), matching upstream's !isDirectoryDep clause in depIsPresent at lockfileToDepGraph.ts:226-228. Without this the mutable source dir would never be re-walked on a second install.

  3. DirectoryFetch classified as fetch-side failure (create_virtual_store.rs:1041-1051) — added to the is_fetch_side_failure match arm, matching upstream's catch at lockfileToDepGraph.ts:286-298, which wraps the whole fetchPackage dispatch and silently swallows directory-fetcher errors on optional snapshots.

Tests: 15 directory-fetcher tests pass, 268 package-manager unit tests pass, clippy clean.


Written by an agent (Claude Code, claude-opus-4-7).

… macro

Dylint's `perfectionist::macro-trailing-comma` lint flagged the
`matches!` invocation added in f2db87c. Add the trailing comma to
clear the lint.

CI: https://github.com/pnpm/pnpm/actions/runs/25958605314/job/76309781526

---
Written by an agent (Claude Code, claude-opus-4-7).
Copilot AI review requested due to automatic review settings May 16, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@zkochan zkochan merged commit 29a42ef into main May 16, 2026
31 of 32 checks passed
@zkochan zkochan deleted the directory-fetcher-port branch May 16, 2026 10:05
github-actions Bot pushed a commit to Eyalm321/pnpm that referenced this pull request May 18, 2026
…pm#11678)

* feat(pacquet/directory-fetcher,pacquet/package-manager): port directory fetcher for injected workspace deps

Port pnpm's `fetching/directory-fetcher` package as a new
`pacquet-directory-fetcher` crate and wire `LockfileResolution::Directory`
into `InstallPackageBySnapshot::run` so injected workspace deps
(`file:./local-pkg` + `dependenciesMeta[*].injected = true`) actually
install instead of returning `UnsupportedResolution`.

Mirrors upstream's
[`fetching/directory-fetcher/src/index.ts`](https://github.com/pnpm/pnpm/blob/85ceff2383/fetching/directory-fetcher/src/index.ts):

- `walk_all_files` ports `fetchAllFilesFromDir` — recursive walk,
  `node_modules` exclusion at any depth, broken-symlink ENOENT skip,
  `resolve_symlinks` toggle between `fileStat` (`fs::metadata`) and
  `realFileStat` (`symlink_metadata` + `canonicalize`).
- `walk_package_files` ports `fetchPackageFilesFromDir` — delegates
  to `pacquet_git_fetcher::packlist` for the npm-packlist filter,
  tolerates a missing manifest for the Bit-workspace shape upstream
  documents at L63-L66.
- `DirectoryFetcher::run` returns `files_map` / `manifest` /
  `requires_build`, matching upstream's `FetchResult` minus
  `local: true` (implicit at the call site) and `packageImportMethod`
  (encoded by which slot the install dispatcher writes to).

Install dispatch:

- `install_package_by_snapshot.rs` resolves `dir_resolution.directory`
  against the newly threaded `workspace_root` (upstream's
  `lockfileDir`), calls `DirectoryFetcher::run`, and hands the
  source-path `files_map` straight through as `cas_paths`. The
  existing `import_indexed_dir` / `link_file` path accepts arbitrary
  source paths so no CAFS write is needed — directory deps don't go
  through the store at all, matching upstream's
  `local: true, packageImportMethod: 'hardlink'` semantics.
- `create_virtual_store.rs`'s `snapshot_cache_key` now returns
  `Ok(None)` for directory resolutions: there's no warm-cache key
  to recover from (the source dir may have changed since last
  install), so every install re-walks. Matches upstream.

Deferred follow-ups (out of scope for this PR):

- `resolveSymlinksInInjectedDirs` / `includeOnlyPackageFiles`
  config-knob plumbing — the dispatch hard-codes the upstream
  defaults (`false` / `false`) from
  [`extendInstallOptions.ts:41`](https://github.com/pnpm/pnpm/blob/85ceff2383/installing/deps-installer/src/install/extendInstallOptions.ts#L41)
  for now.
- Injected-dep re-mirror pass — `hoisted_dep_graph.rs` already
  records `injection_targets_by_dep_path` for every directory-typed
  snapshot location, but the post-install mirror that rebuilds those
  copies is not implemented yet.
- `package.json5` / `package.yaml` manifest read — pacquet's
  `safe_read_package_json_from_dir` only handles `package.json`; the
  other two variants upstream's `safeReadProjectManifestOnly`
  supports are a parity gap that affects directory deps the same way
  it affects every other manifest read in pacquet.

---
Written by an agent (Claude Code, claude-opus-4-7).

* fix(pacquet/directory-fetcher,pacquet/package-manager): address CodeRabbit review on pnpm#11678

Three parity / hardening fixes flagged in the CodeRabbit review on PR
pnpm#11678:

1. **Carve out directory snapshots from the current-lockfile-skip
   gate** (`create_virtual_store.rs`). Directory-typed snapshots have
   `integrity() == None`; without the carve-out `integrity_equal`
   collapsed `None == None == true` and the skip filter dropped the
   snapshot whenever a slot for it existed on disk, so a second
   install never re-walked the (mutable) source dir. Mirrors pnpm's
   `!isDirectoryDep` clause in `depIsPresent` at
   <https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L226-L228>.

2. **Add `DirectoryFetch` to `is_fetch_side_failure`**
   (`create_virtual_store.rs`). Upstream's catch at
   [`lockfileToDepGraph.ts:286-298`](https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L286-L298)
   wraps the whole `fetchPackage` dispatch, so directory-fetcher
   errors on optional snapshots are swallowed uniformly with tarball
   / git fetch errors. Without this, an optional injected-directory
   dep whose source was missing would hard-fail the install instead
   of being dropped.

3. **Symlink-cycle guard in `walk_all_inner`** (`directory-fetcher/
   src/walker.rs`). A `loop -> .` (or any ancestor-pointing) symlink
   previously sank the walker into infinite recursion until either
   ENAMETOOLONG or stack overflow fired. Skip-on-revisit keyed off
   `fs::canonicalize`, matching the pattern
   `pacquet_git_fetcher::packlist` already uses for
   `bundleDependencies` cycles. Pnpm's directory-fetcher has the same
   vulnerability; the guard is a defensible divergence because the
   positive-case behavior is identical to pnpm and the cycle case
   degrades from "crash" to "skip with a `tracing::warn`".

Added a regression test
(`walk_all_files_terminates_on_symlink_cycle`) that points a
`loop -> root` symlink at the walk root and asserts the cycle guard
short-circuits before any `loop/` descendant is recorded.

---
Written by an agent (Claude Code, claude-opus-4-7).

* style(pacquet/package-manager): trailing comma on multi-line matches! macro

Dylint's `perfectionist::macro-trailing-comma` lint flagged the
`matches!` invocation added in f2db87c. Add the trailing comma to
clear the lint.

CI: https://github.com/pnpm/pnpm/actions/runs/25958605314/job/76309781526

---
Written by an agent (Claude Code, claude-opus-4-7).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants