Skip to content

ci(benchmark): consolidate#11741

Merged
zkochan merged 18 commits into
mainfrom
claude/consolidate-benchmarks-1qdcn
May 21, 2026
Merged

ci(benchmark): consolidate#11741
zkochan merged 18 commits into
mainfrom
claude/consolidate-benchmarks-1qdcn

Conversation

@KSXGitHub

@KSXGitHub KSXGitHub commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Unify the two install-benchmark stacks (benchmarks/bench.sh and pacquet/tasks/integrated-benchmark/) into one shared Rust orchestrator. Scenario, fixture, workspace-manifest, install-script, and report generation live in one place so changes propagate to both stacks.

Scenario sets per workflow are unchanged: benchmark.yml keeps measuring the same 6 scenarios; pacquet-integrated-benchmark.yml keeps measuring the same 2. Both verdaccio and live-npm registry modes are preserved; neither is removed in favor of the other. All six scenarios accept both pacquet@<rev> and pnpm@<rev> targets.

See the per-commit breakdown in GitHub's commits tab.

Test plan

  • cargo check --locked --workspace --all-targets clean
  • cargo clippy --locked --workspace --all-targets -- -D warnings clean
  • cargo fmt --all -- --check clean
  • Dylint (perfectionist::*) clean
  • cspell clean (rustup added to project dictionary)
  • pacquet-integrated-benchmark.yml measured the same 2 scenarios on the latest commit and posted a no-regression report
  • benchmark.yml measures the same 6 scenarios end-to-end — needs a manual workflow_dispatch run to verify

Written by an agent (Claude Code).

Summary by CodeRabbit

  • New Features

    • Richer benchmark scenarios, explicit target selection (pacquet/pnpm), selectable registry modes, built-in pnpm comparison, and prebuilt/shared benchmark targets with optional GVS warm-up.
  • Documentation

    • Rewritten benchmarking guide and contributor instructions; new wrapper usage and updated scenario/fixture descriptions.
  • Chores

    • Updated CI benchmark workflow behavior (concurrency/toolchain changes) and consolidated benchmarking into a unified runner; removed legacy benchmark scripts and deprecated configs.

Review Change Stack

`pacquet/benchmark/` (the `add.sh` hyperfine script plus its `.npmrc`,
`.yarnrc.yml`, and `bunfig.toml` companions) compared `pacquet add
fastify` against pnpm/yarn/bun's `add fastify`. It was imported once
with the pacquet merge and never wired into any CI workflow, justfile
recipe, README, or other script — `grep -r pacquet/benchmark` returns
nothing outside the files themselves.

Cross-package-manager install comparisons are already covered (and
more rigorously) by `pacquet/tasks/integrated-benchmark` via its
`--with-pnpm` flag, which is what every active PR uses today. The
three remaining benchmark systems (the TS pnpm hyperfine bench under
`benchmarks/`, and the Rust `integrated-benchmark` + `micro-benchmark`
under `pacquet/tasks/`) cover distinct audiences and layers and are
left as-is.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
@coderabbitai

coderabbitai Bot commented May 19, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR replaces the revisions-list benchmark model with a TargetSpec/RegistryMode model, implements a Rust integrated-benchmark orchestrator, rewrites the wrapper script to call it, and updates verification, WorkEnv, CI workflows, and docs to the new CLI and registry modes.

Changes

Integrated Benchmark Refactoring

Layer / File(s) Summary
CLI argument parsing and domain model
pacquet/tasks/integrated-benchmark/src/cli_args.rs
Introduces TargetSpec/TargetKind parsing from pacquet@<rev> / pnpm@<rev>, adds RegistryMode (Verdaccio, Npm, Virtual), expands BenchmarkScenario (adds Peek, FullResolution, GvsWarm), replaces revisions/verdaccio flags with targets/registry, and adds a Cleanup struct plus unit tests for TargetSpec parsing.
Repository verification for multi-target setup
pacquet/tasks/integrated-benchmark/src/verify.rs
Replaces single ensure_git_repo with ensure_pacquet_git_repo and ensure_pnpm_git_repo and updates validate_revision_list to allow @ in revisions while rejecting other disallowed characters.
Main program orchestration
pacquet/tasks/integrated-benchmark/src/main.rs
Parses new CLI into targets, registry_mode, and optional pnpm_repository; canonicalizes repository/work paths; conditionally initializes verdaccio/virtual registry; derives target presence; ensures conditional tool prerequisites; and constructs WorkEnv with targets, registry_mode, and pnpm_repository.
Benchmark environment and execution workflow
pacquet/tasks/integrated-benchmark/src/work_env.rs
Refactors WorkEnv to use targets and registry_mode, derives BenchId from TargetSpec, adds install_command() and sync_bench_repo(), creates per-target artifacts and pristine copies, implements conditional GVS pre-warm, and replaces string-based cleanup with Cleanup + build_cleanup_command().
Benchmark wrapper script
benchmarks/bench.sh
Rewrites the script into a thin wrapper around target/release/integrated-benchmark: validates tools, builds the binary, ensures a local main ref, pre-builds pnpm@HEAD/pnpm@main, iterates scenarios, copies per-scenario artifacts, and generates a consolidated markdown results table (extracting mean/stddev with jq fallback).
Documentation and CI workflows
benchmarks/README.md, pacquet/CONTRIBUTING.md, .github/workflows/benchmark.yml, .github/workflows/pacquet-integrated-benchmark.yml, cspell.json
Updates README to the wrapper-based workflow and new scenarios, revises CONTRIBUTING benchmarking examples, updates GitHub Actions to use ./.github/actions/rustup and ./.github/actions/binstall for hyperfine@1.18.0, changes revision selectors to pacquet@HEAD/pacquet@main (and pnpm@HEAD/pnpm@main), switches certain flags to --registry=verdaccio, and adds rustup to the cspell word list.

Sequence Diagram

sequenceDiagram
  participant CLI as CliArgs
  participant WorkEnv as WorkEnv
  participant Git as Git
  participant Registry as Verdaccio/Virtual/Npm
  participant Runner as integrated-benchmark

  CLI->>WorkEnv: construct(targets, registry_mode, pnpm_repository)
  WorkEnv->>Git: sync_bench_repo(target commit)
  WorkEnv->>Registry: spawn/ensure when Verdaccio/Virtual
  WorkEnv->>Runner: write install scripts & prepare cleanup
  Runner->>WorkEnv: run timed installs, emit BENCHMARK_REPORT(.md/.json)
  WorkEnv->>Git: restore pristine files between iterations
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

  • zkochan

"🐰
I stitched targets, revisions, and a registry in rhyme,
Rust steers the runs while bash marks the time,
Scenarios lined up, reports saved with a cheer,
Syncs, warms, and restores — the results appear,
Benchmarks hop forward — metrics clear!"

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Title check ❓ Inconclusive The title 'ci(benchmark): consolidate' is vague and lacks specificity about the main changes in the pull request. Consider revising to be more descriptive, such as 'ci(benchmark): unify install-benchmark orchestration' or 'ci(benchmark): consolidate Rust benchmark runner' to clearly convey the primary objective of consolidating two benchmark stacks into a single orchestrator.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/consolidate-benchmarks-1qdcn

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@KSXGitHub KSXGitHub closed this May 19, 2026
@KSXGitHub KSXGitHub deleted the claude/consolidate-benchmarks-1qdcn branch May 19, 2026 08:17
@KSXGitHub KSXGitHub restored the claude/consolidate-benchmarks-1qdcn branch May 19, 2026 08:17
@KSXGitHub KSXGitHub reopened this May 19, 2026
@KSXGitHub KSXGitHub changed the title chore(pacquet): drop dormant pacquet/benchmark/add.sh ci(benchmark): consolidate May 19, 2026
claude added 2 commits May 19, 2026 08:48
Generalize the Rust orchestrator at `pacquet/tasks/integrated-benchmark/`
so it can build and benchmark pnpm revisions alongside pacquet
revisions, preparing the ground for `benchmarks/bench.sh` to become a
thin wrapper that delegates here. Existing CI invocations stay
back-compat.

What this commit changes:

- `BenchmarkScenario` gains three pnpm-only variants — `Peek`,
  `FullResolution`, `GvsWarm` — alongside the existing
  `CleanInstall`, `FrozenLockfile`, `FrozenLockfileHotCache`. Each
  scenario declares whether pacquet supports it; pacquet-unsupported
  scenarios reject pacquet targets at startup with a pointed error.
- Per-iteration cleanup grows from a `&[&str]` of paths-to-remove
  into a `Cleanup { remove, restore }` pair. `restore` overwrites
  `pnpm-lock.yaml` and `package.json` from pristine copies saved
  during `init()`, mirroring the defensive copies `bench.sh` does for
  the same scenarios. `--prepare` becomes
  `rm -rf … && cp .saved-X X && cp .saved-Y Y` per bench dir.
- Positional revisions are now `TargetSpec`s parsed from `<rev>`,
  `pacquet@<rev>`, or `pnpm@<rev>`. Bare revisions default to
  pacquet, so the existing CI invocation
  `--scenario=frozen-lockfile --verdaccio --with-pnpm HEAD main`
  parses unchanged.
- For pnpm targets, the build runs `pnpm install && pnpm run compile`
  in the cloned source and resolves the binary as
  `pnpm/dist/pnpm.mjs` (falling back to `.cjs`), matching
  `bench.sh:55-62`. The install script becomes `node <bundle>
  install …` for pnpm targets vs. `./pacquet/target/release/pacquet
  install …` for pacquet targets. `--pnpm-repository` overrides the
  pnpm checkout source; defaults to `--repository` so a monorepo
  checkout works without an extra flag.
- New `--registry={verdaccio,npm,virtual}` flag picks the registry
  the install hits. `verdaccio` keeps the existing
  spawn-or-attach-verdaccio path; `npm` skips the proxy entirely and
  points `.npmrc` at `https://registry.npmjs.org/` for live
  benchmarks; `virtual` (the default) keeps the existing mockito
  passthrough. The old `--verdaccio` short flag is a back-compat
  alias for `--registry=verdaccio`.
- `GvsWarm` flips `enableGlobalVirtualStore: true` in the synthesized
  workspace manifest and runs a pre-hyperfine warm-up of each target
  so timed iterations see a hot GVS.
- `verify::ensure_program` is now target-kind aware: `cargo` is only
  required when a pacquet target is present, `node`/`pnpm` only when
  a pnpm target (or `--with-pnpm`) is present.

What this commit does NOT change:

- The CI workflows (`.github/workflows/benchmark.yml`,
  `.github/workflows/pacquet-integrated-benchmark.yml`) are
  untouched. Their scenario sets stay identical (6 and 2
  respectively) — those land in follow-up commits.
- `benchmarks/bench.sh` is untouched in this commit; the wrapper
  rewrite is a follow-up.
- No new third-party crate dependencies.

Verification:

- `cargo check --bin=integrated-benchmark` clean
- `cargo clippy --bin=integrated-benchmark -- --deny warnings` clean
- `cargo fmt --check -p pacquet-integrated-benchmark` clean
- `cargo check --workspace --all-targets` clean
- `cargo run --bin=integrated-benchmark -- --help` lists all six
  scenarios, the three registry modes, and the new `--pnpm-repository`
  flag; positional `<TARGETS>...` doc reflects the new spec syntax.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
The `derive_ordering` rule (configured in `dylint.toml`) requires every
`#[derive(...)]` attribute to list prefix traits in canonical order
first, then non-prefix traits alphabetically. `RegistryMode` had
`ValueEnum` interleaved between `Copy` and `PartialEq` — moving
`ValueEnum` after `Eq` puts the prefix group back in one run and
makes dylint happy.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
@github-actions

github-actions Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Micro-Benchmark Results

Linux

group                          main                                   pr
-----                          ----                                   --
tarball/download_dependency    1.00      7.6±0.32ms   567.2 KB/sec    1.00      7.6±0.10ms   569.7 KB/sec

@codecov-commenter

codecov-commenter commented May 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.39%. Comparing base (69f8ea8) to head (004940a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #11741   +/-   ##
=======================================
  Coverage   87.39%   87.39%           
=======================================
  Files         200      200           
  Lines       23514    23514           
=======================================
+ Hits        20550    20551    +1     
+ Misses       2964     2963    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ints

The dylint job runs the `perfectionist` lint family with `-D warnings`,
which forbids single-letter function parameters, generic parameters,
and closure parameters. The previous commit introduced five
violations:

- `TargetSpec::from_str(s: &str)` — single-letter fn param `s` →
  rename to `input`.
- `build_cleanup_command<'a, I, BenchDir>` — single-letter generic
  `I` → rename to `Ids` (a noun matching the parameter's role).
- Three `|t|` closures in `main.rs` that filter/map over `targets` →
  rename to `|target|`.

Pure rename, no behavior change. Verified `cargo clippy --bin=integrated-benchmark
-- --deny warnings` and `cargo fmt --check` clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
@github-actions

github-actions Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Integrated-Benchmark Report (Linux)

Scenario: Frozen Lockfile

Command Mean [s] Min [s] Max [s] Relative
pacquet@HEAD 2.563 ± 0.195 2.407 3.047 1.04 ± 0.09
pacquet@main 2.454 ± 0.084 2.349 2.604 1.00
pnpm 4.904 ± 0.039 4.833 4.960 2.00 ± 0.07
BENCHMARK_REPORT.json
{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 2.5632043331800003,
      "stddev": 0.19493267113395893,
      "median": 2.5025174152800003,
      "user": 2.7422587399999996,
      "system": 3.8066402599999996,
      "min": 2.4071387422800004,
      "max": 3.04663413928,
      "times": [
        2.68049611528,
        2.4936991932800003,
        3.04663413928,
        2.6665205702800003,
        2.43888659628,
        2.5238352342800003,
        2.41745053228,
        2.44604657128,
        2.51133563728,
        2.4071387422800004
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 2.4537371118799998,
      "stddev": 0.08425808498888249,
      "median": 2.4366384832800003,
      "user": 2.75710094,
      "system": 3.754935259999999,
      "min": 2.3485786932800004,
      "max": 2.6036598752800004,
      "times": [
        2.3485786932800004,
        2.5472386842800003,
        2.39222925928,
        2.4045135662800003,
        2.6036598752800004,
        2.37755737128,
        2.49865152928,
        2.4687634002800003,
        2.50850765128,
        2.3876710882800003
      ]
    },
    {
      "command": "pnpm",
      "mean": 4.904130496980001,
      "stddev": 0.038577557748176804,
      "median": 4.90602158828,
      "user": 8.309343639999998,
      "system": 4.18000916,
      "min": 4.83300584728,
      "max": 4.96001776528,
      "times": [
        4.87314659128,
        4.93718390228,
        4.90830900128,
        4.83300584728,
        4.89990355828,
        4.903734175279999,
        4.91728991528,
        4.96001776528,
        4.9427671452799995,
        4.86594706828
      ]
    }
  ]
}

Scenario: Frozen Lockfile (Hot Cache)

Command Mean [ms] Min [ms] Max [ms] Relative
pacquet@HEAD 702.9 ± 40.1 680.4 815.9 1.00
pacquet@main 725.3 ± 39.8 691.6 829.6 1.03 ± 0.08
pnpm 2649.0 ± 46.4 2580.8 2724.4 3.77 ± 0.22
BENCHMARK_REPORT.json
{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 0.7029015570799999,
      "stddev": 0.04010473417876268,
      "median": 0.69195584408,
      "user": 0.38850977999999997,
      "system": 1.5677578799999998,
      "min": 0.6804301230800001,
      "max": 0.8158909000800001,
      "times": [
        0.8158909000800001,
        0.69498806108,
        0.69458789708,
        0.6873159250800001,
        0.68881001308,
        0.6834362000800001,
        0.6893237910800001,
        0.6804301230800001,
        0.69763251808,
        0.6966001420800001
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 0.72525979578,
      "stddev": 0.03981585769840402,
      "median": 0.7110447520800001,
      "user": 0.39156968,
      "system": 1.58550228,
      "min": 0.69156643208,
      "max": 0.82964211308,
      "times": [
        0.82964211308,
        0.71485875608,
        0.70288337708,
        0.69156643208,
        0.7072307480800001,
        0.7399505280800001,
        0.7064100940800001,
        0.70057545208,
        0.72332597708,
        0.7361544800800001
      ]
    },
    {
      "command": "pnpm",
      "mean": 2.648976157080001,
      "stddev": 0.0464023313941789,
      "median": 2.6571728830800003,
      "user": 3.28831018,
      "system": 2.20972878,
      "min": 2.58083468308,
      "max": 2.72443879208,
      "times": [
        2.67243221208,
        2.66298893208,
        2.69369458108,
        2.65135683408,
        2.58083468308,
        2.6785996340800002,
        2.61555660108,
        2.59053935108,
        2.72443879208,
        2.61931995008
      ]
    }
  ]
}

Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
Comment thread pacquet/tasks/integrated-benchmark/src/cli_args.rs Outdated
claude added 2 commits May 19, 2026 09:37
Clap surfaces field rustdocs verbatim as `--help` text, so verbose
rustdocs that read like architecture notes blow up the help output.
Per review on #11741, shorten each clap-visible doc to one
line / one short paragraph that reads cleanly as CLI help:

- `--verdaccio`: drop the "kept for back-compat" historical note —
  not useful to a user reading help. The compat behaviour is still
  the same (`-V` is a short alias and overrides `--registry`).
- `--pnpm-repository`: collapse the rationale ("monorepo checkout
  containing both works out of the box") into the existing
  "Defaults to `--repository`" line; the rationale is for code
  readers, not CLI users.
- `--with-pnpm`: collapse to one line — "Also benchmark the
  system-installed pnpm." "System-installed" disambiguates from the
  new `pnpm@<rev>` target syntax (a CLI user could otherwise wonder
  whether `--with-pnpm` and `pnpm@HEAD` are equivalent), so it
  stays in the wording even at one line.
- `targets`: collapse the bulleted variant list into one inline
  enumeration. The exhaustive bullets came out poorly in clap's
  help renderer anyway (run-together dashes).
- `RegistryMode` / `BenchmarkScenario` value-enum variants: one
  line each. The verbose verdaccio explanation, the "frozen
  lockfile, warm local store" phrasing, the "no lockfile and
  without local cache" expansion etc. are all redundant with the
  scenario / mode names. Keep the "Pnpm-only" marker on the three
  pnpm-only scenarios — that's the load-bearing detail.

`Cleanup` struct docs (`Cleanup` itself is not clap-visible, used
internally) are also trimmed in the same pass, per the same review
question on line 152: the `restore` field doc explained `(dst,
src)` semantics three times. Reduced to one line plus a brief "why"
clause.

Verified `cargo run --bin=integrated-benchmark -- --help` renders
each flag and value-enum variant on one line; `cargo clippy
--deny warnings` and `cargo fmt --check` still clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
bench.sh becomes a thin shell wrapper that delegates to the shared Rust
orchestrator at `pacquet/tasks/integrated-benchmark/`. Scenario set,
registry choice (`--registry=npm` — live npmjs.com, no proxy), warmup /
runs env-var contract, and the `Temp directory kept at: <dir>` line
benchmark.yml parses are all preserved.

Layout changes:

- `benchmarks/fixture.package.json` → `benchmarks/fixture/package.json`
  so the orchestrator's `--fixture-dir` (which expects a directory with
  a `package.json` and an optional `pnpm-lock.yaml`) can be pointed at
  it directly.
- New `benchmarks/fixture/pnpm-lock.yaml` (12 144 lines, 416 KB,
  generated once with `pnpm install --lockfile-only` against live npm).
  Committing the lockfile pins the resolution graph across CI runs and
  avoids per-revision lockfile drift that would otherwise change what
  the four lockfile-bearing scenarios measure. Same trade-off the
  pacquet integrated-benchmark has made since its initial import.
- `benchmarks/generate-results.js` deleted — hyperfine's own
  `--export-markdown` output (which the Rust orchestrator already
  passes through) replaces the bespoke renderer. `bench.sh` adds a
  small jq-driven consolidated table on top for the existing
  `results.md` filename the comment-poster step in `benchmark.yml`
  reads.

The script:

1. Builds `integrated-benchmark` in release mode.
2. Runs `--build-only pnpm@HEAD pnpm@main` once to clone+build both
   pnpm revisions (`pnpm install && pnpm run compile`) into the
   shared work-env. Subsequent per-scenario invocations reuse the
   built bundles — `pnpm install` is a no-op and `tsgo --build` is
   incremental, so the rebuild overhead is small.
3. Invokes the orchestrator once per scenario for the six scenarios
   bench.sh measured before (frozen-lockfile-hot-cache, peek,
   full-resolution, frozen-lockfile, clean-install, gvs-warm),
   producing per-scenario `BENCHMARK_REPORT.md` / `.json` and a
   consolidated `results.md`.

Workflow update (`benchmark.yml`):

- Replace the wget + dpkg hyperfine install with the existing
  `./.github/actions/binstall` action (same binary version, 1.18.0;
  matches the pacquet workflows).
- Add `./.github/actions/rustup` to bootstrap the Rust toolchain the
  orchestrator needs. Uses a dedicated `shared-key: pnpm-benchmark`
  cache so the bench cache doesn't churn the pacquet ones.
- Drop the explicit `pnpm run compile` step — the orchestrator runs
  it per revision inside the work-env, against the revision's own
  source. (The repo-root bundle would have been wrong for `main`
  anyway, since the checkout is at `HEAD`.)

Scenario set unchanged (still 6).

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
Comment on lines -60 to -61
- name: Compile
run: pnpm run compile

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code,

Can you explain why the step pnpm run compile was removed here?

Before this PR, what was the purpose of benchmark.yml CI? What did it measure? Did it measure some existing pnpm? Or did it measure the pnpm compiled from this codebase?

Unless it was moved elsewhere and I didn't notice it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question — the compile work was moved, not dropped, but the prose in my commit message glossed over it. Let me make it explicit.

Before this PR. benchmark.yml's pnpm run compile at the repo root produced pnpm/dist/pnpm.mjs for whatever was checked out (= the PR head). benchmarks/bench.sh then resolved that bundle as the "branch" binary, and built main itself by creating a git worktree at ../.pnpm-bench-main, running pnpm install && pnpm run compile inside it, and resolving its pnpm/dist/pnpm.mjs as the "main" binary. So the workflow measured pnpm compiled from this codebase, HEAD vs main.

After this PR. The same measurement, by the same method, but the compile moved into the orchestrator's per-revision build path. For a pnpm@<rev> target the orchestrator does (work_env.rs#L203-226):

fn build_pnpm(&self, revision: &str) {
    // ... resolve_revision, sync_bench_repo (git clone/fetch + checkout)
    Command::new("pnpm")
        .current_dir(&revision_repo)
        .arg("install")
        .pipe(executor("pnpm install"));
    Command::new("pnpm")
        .current_dir(&revision_repo)
        .arg("run").arg("compile")
        .pipe(executor("pnpm run compile"));
}

bench.sh invokes the orchestrator once up front with --build-only pnpm@HEAD pnpm@main, which fires build_pnpm for both revisions. Each ends up at bench-work-env/pnpm@HEAD/pnpm-source/pnpm/dist/pnpm.mjs and .../pnpm@main/pnpm-source/pnpm/dist/pnpm.mjs — the same bundle the install script then loads via node <path> install ….

So the repo-root compile is now redundant: it would have produced a pnpm/dist/pnpm.mjs that nothing reads — the orchestrator works against its own per-revision clones, not the repo root.

The measurement contract is unchanged: still HEAD vs main, still pnpm compiled from this codebase, still six scenarios. The build just moved into one consistent path (the orchestrator) instead of being split between two (benchmark.yml's compile for HEAD, bench.sh's worktree compile for main).

I'll rewrite that sentence in the commit message in a follow-up commit so the diff isn't misleading on its own.


Generated by Claude Code

Comment thread benchmarks/generate-results.js Outdated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code,

So I guess the task of this file was moved elsewhere?

To where exactly though? The bash script (bench.sh) or the Rust programs at pacquet/tasks/?

Can you explain why you moved it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file did three things; here's where each part lives now:

  1. Read each per-scenario hyperfine JSON and render its mean ± stddev as markdown. — Moved into the orchestrator (Rust). The Rust orchestrator's benchmark() already passes --export-markdown=BENCHMARK_REPORT.md to hyperfine (work_env.rs benchmark() end), so hyperfine itself produces the per-scenario markdown table that this script used to reconstruct from JSON. Same content; one fewer rendering step.

  2. Concatenate per-scenario tables into a single results.md. — Moved into bench.sh (Bash). Specifically, the loop near the bottom that appends | $i | $label | $main_cell | $head_cell | rows. It pulls mean ± stddev per variant out of each BENCHMARK_REPORT.json via jq. That's the only thing that didn't have an upstream home in the orchestrator — it's a pnpm-bench-specific summary (the pacquet workflow uses a different layout with one detail-section per scenario, not a single comparison table).

  3. Pretty number formatting (3-decimal trimming). — Moved into bench.sh's jq expression: (.mean*1000|floor)/1000. Same precision as the JS toFixed(3) did, just expressed in jq.

Why moved: with the orchestrator now responsible for invoking hyperfine for the pnpm bench, hyperfine's own --export-markdown output is already in the work-env. Reading the JSON again from JS and re-rendering it was duplicating what hyperfine had already done. So part (1) folds upstream; parts (2) and (3) are tiny enough that a jq loop in bench.sh is cheaper than keeping a separate Node script in the dependency surface.

If you'd rather keep a JS renderer for clarity / cross-platform reasons, happy to revert this part and have bench.sh call back into a Node script for the consolidated table — let me know.


Generated by Claude Code

claude added 6 commits May 20, 2026 00:06
Round of fixes raised by my own /review pass. None affect what the
benchmarks measure — they tighten contracts the previous commits left
loose, drop dead init work, and correct three small documentation /
comment errors.

- `verify::ensure_git_repo` was checked unconditionally and asserted
  `Cargo.toml` + `Cargo.lock` exist. That's a pacquet-shape check;
  pointing `--pnpm-repository` at a standalone pnpm clone would panic.
  Split into `ensure_pacquet_git_repo` and `ensure_pnpm_git_repo`
  (the latter looks for `pnpm/package.json` or `pnpm-workspace.yaml`)
  and call each only for the target kinds that need them.
- `pnpm` was only added to the ensure-program list when a pnpm target
  or `--with-pnpm` was in play. The proxy-cache populator that runs
  for verdaccio/virtual modes also shells out to `pnpm install`, so a
  pacquet-only run with `--registry=virtual` (the default) could miss
  the prerequisite check. Require `pnpm` for any non-npm registry mode.
- `validate_revision_list`'s allowed alphabet rejected `@` and `/`,
  even though `TargetSpec::from_str`'s "unknown prefix keeps full
  string as pacquet rev" branch (and its unit test) explicitly accepts
  revs like `v1.0.0@sha`. Widen the alphabet so the documented
  contract holds end-to-end. `@` and `/` are not shell metacharacters
  in the contexts we pass revs to (git CLI args, bench-dir names).
- `init()` was synthesizing files for the `.init-proxy-cache` bench
  dir regardless of registry mode. With `--registry=npm` those files
  are never read. Gate the whole `INIT_PROXY_CACHE` entry in the
  init id-list on `populate_proxy_cache` so npm runs skip the work
  entirely.
- `--registry-port` doc: note that it's ignored when `--registry=npm`.
- `benchmarks/bench.sh`: defensive `git fetch origin main:main` so
  `pnpm@main` resolves on dispatch runs where `actions/checkout` only
  created a local ref for the head branch. Move the `read_cell`
  helper above the scenario loop instead of redefining it every
  iteration. Switch jq from `floor` to `round` so cell values match
  the rounding the deleted `generate-results.js` did with
  `toFixed(3)`. Drop dead `${main_cell:-n/a}` fallbacks (read_cell
  always returns a non-empty string). Rewrite the misleading
  "the per-scenario build step is a no-op" comment — the bundle
  step is not incremental and re-runs each time. Six extra bundle
  runs is accepted overhead in exchange for one consistent build
  path shared with the pacquet bench.
- `benchmarks/README.md`: drop the false "Rust toolchain is
  bootstrapped automatically by `cargo build`" sentence.

Verified `cargo clippy --bin=integrated-benchmark -- --deny warnings`
and `cargo fmt --check -p pacquet-integrated-benchmark` clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
…ings

Per direct feedback: `--verdaccio` was redundant with `--registry=verdaccio`
and the back-compat justification was bogus for an internal tool whose
callers I own. Removed the flag, the back-compat short-circuit method, and
updated `pacquet-integrated-benchmark.yml` to pass `--registry=verdaccio`
in both scenario steps.

Iteration-2 review fixes (none affect what the benchmarks measure):

- `bench.sh` fetch fallback was a `||`-chain that swallowed every error
  (no-network, missing remote, permission-denied) and left the next
  `git rev-parse main` failing with an opaque message. Skip the fetch
  when `refs/heads/main` already exists locally; otherwise run the
  fetch in the foreground so its real error surfaces.
- `read_cell` jq filter returned an empty string when the requested
  target wasn't in `BENCHMARK_REPORT.json` (the `select` produced no
  rows; jq exited 0). Wrap the filter in `[…] | first // "n/a"` so the
  missing-target case renders as `n/a` in the results table.
- `validate_revision_list` had `/` in the allowed alphabet from
  iteration 1. A target like `pnpm@origin/main` would create a nested
  bench dir (`pnpm@origin/main/`) — confusing layout and a slash in
  the hyperfine command name. Drop `/` and document that callers
  pre-resolve remote-tracking refs to a local branch or SHA. `bench.sh`
  already does this via the `git fetch origin main:main` step above.
- `benchmark.yml` had no concurrency group: two parallel
  `workflow_dispatch` runs against the same PR would post duplicate
  comments. Add `concurrency: { group: benchmark-${{ pr_number || ref
  }}, cancel-in-progress: false }`. Don't cancel-in-progress — killing
  a long bench mid-run wastes minutes and produces no data.
- `sync_bench_repo` didn't reset the worktree before the next
  `git checkout <sha>`. After a pnpm build, `pnpm install` may have
  rewritten `pnpm-lock.yaml` in the tracked tree, and subsequent
  scenarios' checkouts fail with "Your local changes would be
  overwritten." Run `git reset --hard` before `checkout` whenever the
  bench repo existed before this invocation. The fresh-clone path is
  untouched.

CI fix:

- `cspell` rejected `rustup` in `benchmarks/README.md` (the iteration-1
  rephrase added "via [rustup](https://rustup.rs)"). Added `rustup` to
  the project dictionary in `cspell.json` between `rushstack` and
  `safecrlf` to keep the list sorted.

Verified `cargo clippy --bin=integrated-benchmark -- --deny warnings`,
`cargo fmt --check -p pacquet-integrated-benchmark`, `bash -n
benchmarks/bench.sh`, and `cspell benchmarks/README.md` all clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
Iteration-2 added `git reset --hard` to `sync_bench_repo` so the next
`git checkout <commit>` wouldn't be blocked by tracked files left dirty
by `pnpm install`. The reset was gated on `existed_before` — i.e.
"the bench-dir path existed when we entered". That gate is too wide:
it also matches the "directory exists but `.git` doesn't" case, where
the previous statement just ran `git init --initial-branch=__blank__`
and the freshly-fetched commit is in `FETCH_HEAD` but HEAD is still
unborn. `git reset --hard` against an unborn HEAD fatal-errors.

Tighten the gate to `had_existing_git` (= dir existed AND `.git` was
already present). The init-then-fetch path skips the reset; the
freshly-cloned path also skips it (revision_repo didn't exist); only
the "reuse a prior bench repo" path runs the reset, which is where
worktree dirt actually accumulates.

Raised by iteration-3 self-review.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
The bare-revision-aliases-pacquet behavior added two failure modes the
parser had to handle separately: an unknown-prefix branch that fell
through to "treat the whole string as a pacquet rev verbatim", and a
test (`target_spec_unknown_prefix_keeps_full_string_as_pacquet_rev`)
that locked that branch in. Both went away once `pacquet@<rev>` and
`pnpm@<rev>` are the only accepted shapes.

`TargetSpec::from_str` collapses to one `split_once('@')` + match:

    pacquet@<rev> | pnpm@<rev>

Anything else returns a precise error (`unknown kind`, `<rev> must
not be empty`, or `must be pacquet@<rev> or pnpm@<rev>`). The doc
on `--targets` and the `--help` text drop the "= `pacquet@<rev>`"
alias note.

Callers updated to the explicit form:

- `.github/workflows/pacquet-integrated-benchmark.yml`: three
  `--build-only HEAD main` / `--with-pnpm HEAD main` invocations
  become `pacquet@HEAD pacquet@main`.
- `pacquet/CONTRIBUTING.md`: four examples updated.
- `benchmarks/bench.sh` already used `pnpm@HEAD pnpm@main` — no
  change needed.

Tests rewritten: `target_spec_bare_revision_is_pacquet` and
`target_spec_unknown_prefix_keeps_full_string_as_pacquet_rev`
removed; `target_spec_unprefixed_is_rejected`,
`target_spec_unknown_prefix_is_rejected`, and
`target_spec_empty_rev_is_rejected` added.

Verified `cargo clippy --bin=integrated-benchmark -- --deny warnings`,
`cargo fmt --check`, and `cargo test -p pacquet-integrated-benchmark`
(5/5 pass).

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
The previous commit landed `let (prefix, rev) = input.split_once('@').ok_or_else(|| { … })?;`
which rustfmt rewrites into a vertical `.split_once / .ok_or_else` chain.
`cargo fmt --check -p pacquet-integrated-benchmark` happened to be clean
locally on the unformatted form, but `cargo fmt --all -- --check`
(what the CI Format job runs) flagged it.

No behavior change.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
@KSXGitHub KSXGitHub marked this pull request as ready for review May 21, 2026 05:54
@KSXGitHub KSXGitHub requested a review from zkochan as a code owner May 21, 2026 05:54
Copilot AI review requested due to automatic review settings May 21, 2026 05:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR consolidates the pnpm and pacquet install benchmark implementations by moving scenario/fixture/workspace/install-script/report generation into a single shared Rust orchestrator (pacquet/tasks/integrated-benchmark). The existing scenario sets per workflow are preserved (pnpm workflow still runs 6 scenarios; pacquet workflow still runs 2).

Changes:

  • Refactors the integrated benchmark orchestrator to support multiple target kinds (pacquet@<rev>, pnpm@<rev>), registry modes, and more robust per-iteration cleanup/restoration.
  • Replaces the pnpm benchmarks/bench.sh logic with a thin wrapper that invokes the shared Rust orchestrator and emits the same consolidated results.md.
  • Updates CI workflows and documentation to use the new target syntax and shared orchestrator; removes legacy benchmark artifacts/scripts.

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
pacquet/tasks/integrated-benchmark/src/work_env.rs Adds multi-target orchestration (pacquet + pnpm), repo sync/build logic, and enhanced cleanup/pristine restore.
pacquet/tasks/integrated-benchmark/src/verify.rs Splits repo verification into pacquet vs pnpm checks; updates revision validation rules.
pacquet/tasks/integrated-benchmark/src/main.rs Wires new CLI options (registry mode, pnpm repo override, targets) and validates scenario/target compatibility.
pacquet/tasks/integrated-benchmark/src/cli_args.rs Introduces TargetSpec parsing (pacquet@… / pnpm@…), registry modes, new scenarios, and cleanup model + tests.
pacquet/CONTRIBUTING.md Updates benchmark command examples to the new pacquet@… target format.
pacquet/benchmark/bunfig.toml Removes legacy benchmark configuration.
pacquet/benchmark/add.sh Removes legacy benchmark script.
pacquet/benchmark/.yarnrc.yml Removes legacy benchmark configuration.
pacquet/benchmark/.npmrc Removes legacy benchmark configuration.
cspell.json Adds rustup to the dictionary.
benchmarks/README.md Updates pnpm benchmark documentation to describe the shared orchestrator-based flow and fixture.
benchmarks/generate-results.js Removes legacy results generator (superseded by orchestrator + wrapper output).
benchmarks/fixture/package.json Adds a committed fixture manifest for pnpm benchmark scenarios.
benchmarks/bench.sh Replaces bespoke benchmark logic with a wrapper invoking integrated-benchmark across 6 scenarios.
.github/workflows/pacquet-integrated-benchmark.yml Updates commands to use pacquet@… targets and new --registry=verdaccio flag.
.github/workflows/benchmark.yml Adds concurrency settings, installs Rust toolchain, switches hyperfine install to the shared binstall action.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +93 to +98
/// Path to the bundled pnpm entry produced by `pnpm run compile`.
/// Mirrors `resolve_pnpm_bin` in `benchmarks/bench.sh` — prefer
/// `pnpm.mjs`, fall back to `pnpm.cjs`.
fn pnpm_bundle_path(source_dir: &Path) -> PathBuf {
let mjs = source_dir.join("pnpm/dist/pnpm.mjs");
if mjs.exists() { mjs } else { source_dir.join("pnpm/dist/pnpm.cjs") }
BenchId::PnpmRevision(rev) => {
let bundle = WorkEnv::pnpm_bundle_path(&self.pnpm_source_dir(rev));
let bundle = bundle.to_str().expect("convert pnpm bundle path to UTF-8");
format!("node {bundle}")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — confirmed and fixed in 2a886bb.

The fix collapses with the pnpm_bundle_path init-time-vs-build-time bug from r3278981782: switching the install command from an absolute, init-time-resolved path to a runtime [ -f X ] check using paths relative to the bench dir (./pnpm-source/pnpm/dist/pnpm.mjs / .cjs). The script cds into that dir at its top, so the relative form resolves correctly and there's no {work_env}/... interpolation at all — no embedded absolute path means no shell-metacharacter risk regardless of what --work-env the user passes.

Resulting install command:

node "$([ -f ./pnpm-source/pnpm/dist/pnpm.mjs ] && echo ./pnpm-source/pnpm/dist/pnpm.mjs || echo ./pnpm-source/pnpm/dist/pnpm.cjs)"

Generated by Claude Code

Comment on lines +62 to +67
// `@` is allowed so `git rev-parse` reflog syntax (`HEAD@{1}`, etc.)
// and tag-with-suffix revisions like `v1.0.0@sha` reach git intact.
// `/` is intentionally excluded: a revision like `origin/main` would
// produce a nested bench-dir path (`pnpm@origin/main/`) — callers
// should pre-resolve remote-tracking refs to a local branch or SHA.
let invalid_char = revision.chars().find(|char| !matches!(char, 'a'..='z' | 'A'..='Z' | '0'..='9' | '-' | '_' | '+' | '.' | '~' | '^' | '@'));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — comment was misleading. Fixed in 2a886bb.

The original claim was that @ is allowed for git's reflog syntax (HEAD@{1}), but the alphabet still rejects { and } so reflog revs don't actually go through. The real reason @ is allowed is v1.0.0@sha-style tag suffixes, not reflog. Updated the comment to say so and to make the { / } exclusion deliberate rather than accidental — { is a shell metacharacter we don't want in bench-dir names anyway, and the bench doesn't claim to support reflog revisions.


Generated by Claude Code

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
benchmarks/bench.sh (1)

23-28: ⚡ Quick win

Consider making jq a required dependency.

The read_cell function (lines 76-84) checks for jq and falls back to "n/a" if it's missing. Without jq, the benchmark runs but produces a table filled with "n/a" values, which is less useful. Adding jq to this upfront validation would fail early with a clear message rather than generating incomplete output.

🔧 Proposed addition
-for tool in cargo hyperfine pnpm node git; do
+for tool in cargo hyperfine pnpm node git jq; do
   if ! command -v "$tool" >/dev/null; then
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmarks/bench.sh` around lines 23 - 28, The script currently validates
required tools in the startup loop but omits jq, while read_cell() later
silently falls back to "n/a"; add "jq" to the validation list in the for tool in
... loop so the script fails early with a clear message if jq is missing (this
affects the startup validation block shown and relates to the read_cell function
used later), and optionally remove or adjust the read_cell fallback to avoid
producing "n/a" rows when jq is absent.
pacquet/tasks/integrated-benchmark/src/work_env.rs (1)

328-334: 💤 Low value

Redundant directory argument to git init.

current_dir is already set to revision_repo, so passing it again as a positional argument to git init is redundant. Not a bug, but the extra arg can be removed.

♻️ Suggested simplification
             Command::new("git")
                 .current_dir(revision_repo)
                 .arg("init")
-                .arg(revision_repo)
                 .arg("--initial-branch=__blank__")
                 .pipe(executor("git init"));
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pacquet/tasks/integrated-benchmark/src/work_env.rs` around lines 328 - 334,
The git invocation in the Command built in work_env.rs is passing the repository
path twice: current_dir(revision_repo) plus .arg(revision_repo) is redundant;
remove the .arg(revision_repo) so the command becomes
Command::new("git").current_dir(revision_repo).arg("init").arg("--initial-branch=__blank__").pipe(executor("git
init")); (refer to the Command::new("git") usage and the
.current_dir(revision_repo)/.arg("init")/.arg("--initial-branch=__blank__") call
chain to locate the code).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pacquet/tasks/integrated-benchmark/src/work_env.rs`:
- Around line 96-99: pnpm_bundle_path() is checking for pnpm.mjs vs pnpm.cjs
during init() (used by install_command()) before build() creates those files, so
it deterministically picks the wrong path; fix by deferring resolution to
runtime or moving script creation after build: either change install_command()
to write an install script that resolves pnpm.mjs vs pnpm.cjs at execution time
(inspect both paths at runtime inside the script) or stop calling
pnpm_bundle_path() during init() and instead call it or generate the install
script after build() completes so the correct pnpm_bundle_path() (pnpm.mjs or
pnpm.cjs) is known.

---

Nitpick comments:
In `@benchmarks/bench.sh`:
- Around line 23-28: The script currently validates required tools in the
startup loop but omits jq, while read_cell() later silently falls back to "n/a";
add "jq" to the validation list in the for tool in ... loop so the script fails
early with a clear message if jq is missing (this affects the startup validation
block shown and relates to the read_cell function used later), and optionally
remove or adjust the read_cell fallback to avoid producing "n/a" rows when jq is
absent.

In `@pacquet/tasks/integrated-benchmark/src/work_env.rs`:
- Around line 328-334: The git invocation in the Command built in work_env.rs is
passing the repository path twice: current_dir(revision_repo) plus
.arg(revision_repo) is redundant; remove the .arg(revision_repo) so the command
becomes
Command::new("git").current_dir(revision_repo).arg("init").arg("--initial-branch=__blank__").pipe(executor("git
init")); (refer to the Command::new("git") usage and the
.current_dir(revision_repo)/.arg("init")/.arg("--initial-branch=__blank__") call
chain to locate the code).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2f55e850-1692-4ab0-88a5-621d0e93f441

📥 Commits

Reviewing files that changed from the base of the PR and between 0fb7233 and 273e4e3.

⛔ Files ignored due to path filters (1)
  • benchmarks/fixture/pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (16)
  • .github/workflows/benchmark.yml
  • .github/workflows/pacquet-integrated-benchmark.yml
  • benchmarks/README.md
  • benchmarks/bench.sh
  • benchmarks/fixture/package.json
  • benchmarks/generate-results.js
  • cspell.json
  • pacquet/CONTRIBUTING.md
  • pacquet/benchmark/.npmrc
  • pacquet/benchmark/.yarnrc.yml
  • pacquet/benchmark/add.sh
  • pacquet/benchmark/bunfig.toml
  • pacquet/tasks/integrated-benchmark/src/cli_args.rs
  • pacquet/tasks/integrated-benchmark/src/main.rs
  • pacquet/tasks/integrated-benchmark/src/verify.rs
  • pacquet/tasks/integrated-benchmark/src/work_env.rs
💤 Files with no reviewable changes (5)
  • pacquet/benchmark/add.sh
  • pacquet/benchmark/.yarnrc.yml
  • pacquet/benchmark/.npmrc
  • benchmarks/generate-results.js
  • pacquet/benchmark/bunfig.toml
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
pacquet/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

pacquet/**/*.rs: When porting a function that fires pnpm:<channel> events through globalLogger, logger.debug(), or streamParser.write(), mirror the call site, payload, and ordering so the reporter parses pacquet's NDJSON the same way it parses pnpm's.
Declare a newtype wrapper for branded string types. Do not collapse the brand into a plain String or &str.
If upstream always validates before construction, validate in pacquet's wrapper too. The wrapper must construct only via TryFrom<String> and/or FromStr. Do not provide an infallible public constructor.
If upstream never validates, just brand for type-safety. Expose an infallible From<String> (and From<&str> when convenient).
If upstream occasionally constructs without validation, expose from_str_unchecked as an escape hatch alongside the validating constructor.
Match upstream serde behavior for branded types that cross JSON, YAML, or INI boundaries. Use #[serde(try_from = "String")] for deserialization and #[serde(into = "String")] for serialization.
Use #[derive(derive_more::From)] and #[derive(derive_more::Into)] for mechanical conversion impls. Fall back to manual impl only when conversion needs custom logic.
String-literal unions should become enums, not newtype wrappers. Model closed sets of valid string values as enums.
Template literal types should be treated as branded strings with validation discipline from rules 2-5.
Choose owned vs. borrowed parameters to minimize copies. Widen to the most encompassing type (&Path over &PathBuf, &str over &String) when it doesn't force extra copies.
Prefer Arc::clone(&x) / Rc::clone(&x) over x.clone() for reference-counted types, so the cost is visible at the call site.
Follow Rust API Guidelines for naming conventions.
Do not use star imports inside module bodies. Write use super::{Foo, bar} instead of use super::*;. Two forms stay allowed: external-crate preludes like use rayon::prelude::*; and root-of-module re-...

Files:

  • pacquet/tasks/integrated-benchmark/src/verify.rs
  • pacquet/tasks/integrated-benchmark/src/main.rs
  • pacquet/tasks/integrated-benchmark/src/cli_args.rs
  • pacquet/tasks/integrated-benchmark/src/work_env.rs
🧠 Learnings (1)
📚 Learning: 2026-05-20T23:07:58.444Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11784
File: pacquet/crates/resolving-deps-resolver/src/hoist_peers.rs:120-133
Timestamp: 2026-05-20T23:07:58.444Z
Learning: When reviewing code in this pacquet Rust port, follow the upstream pnpm compatibility rule: only match pnpm’s behavior exactly. Do not propose review changes that intentionally deviate from pnpm’s documented/observed behavior, even if pnpm appears buggy. If you identify a real bug in pnpm behavior, the review should prioritize fixing it upstream in pnpm first, and avoid implementing a pnpm-behavior workaround here unless the same fix has already landed upstream.

Applied to files:

  • pacquet/tasks/integrated-benchmark/src/verify.rs
  • pacquet/tasks/integrated-benchmark/src/main.rs
  • pacquet/tasks/integrated-benchmark/src/cli_args.rs
  • pacquet/tasks/integrated-benchmark/src/work_env.rs
🪛 actionlint (1.7.12)
.github/workflows/benchmark.yml

[error] 62-62: could not parse action metadata in "/home/jailuser/git/.github/actions/rustup": line 7: unexpected key "type" for definition of input "shared-key"

(action)

🔇 Additional comments (16)
benchmarks/bench.sh (5)

1-16: LGTM!


30-43: LGTM!


45-68: LGTM!


70-85: LGTM!


87-130: LGTM!

pacquet/tasks/integrated-benchmark/src/cli_args.rs (1)

1-308: LGTM!

pacquet/tasks/integrated-benchmark/src/verify.rs (1)

15-41: LGTM!

pacquet/tasks/integrated-benchmark/src/main.rs (1)

1-127: LGTM!

pacquet/tasks/integrated-benchmark/src/work_env.rs (1)

1-34: LGTM!

Also applies to: 36-91, 136-169, 171-233, 235-312, 314-327, 335-595

benchmarks/README.md (1)

1-62: LGTM!

pacquet/CONTRIBUTING.md (1)

143-159: LGTM!

.github/workflows/benchmark.yml (2)

26-30: LGTM!


61-69: LGTM!

.github/workflows/pacquet-integrated-benchmark.yml (2)

138-138: LGTM!


148-148: LGTM!

Also applies to: 162-162

cspell.json (1)

293-293: LGTM!

Comment thread pacquet/tasks/integrated-benchmark/src/work_env.rs Outdated
CodeRabbit / Copilot flagged that `WorkEnv::pnpm_bundle_path()` checked
`pnpm.mjs.exists()` at `init()` time, but the pnpm source dir doesn't
exist yet at that point — `build()` runs after `init()` and is what
clones the repo and runs `pnpm install && pnpm run compile`. So the
existence check was always false at the call site, and the generated
`install.bash` always hardcoded `pnpm.cjs`. Modern pnpm produces both
bundles so the script didn't crash, but the "prefer mjs, fall back to
cjs" intent (mirroring `benchmarks/bench.sh`'s `resolve_pnpm_bin`) was
silently broken — measurements always exercised the CJS startup path.

The fix moves the `[ -f X ]` check into the install script itself, so
it runs after `build()` has produced the bundle:

    node "$([ -f ./pnpm-source/pnpm/dist/pnpm.mjs ] \
            && echo ./pnpm-source/pnpm/dist/pnpm.mjs \
            || echo ./pnpm-source/pnpm/dist/pnpm.cjs)"

Two side wins:

- The path is now relative to the bench dir (which the script `cd`s into
  at the top), so it doesn't depend on a `--work-env` value the orchestrator
  embeds as an absolute prefix. Resolves Copilot's separate concern about
  unquoted absolute paths breaking if the work-env contains shell
  metacharacters.
- `WorkEnv::pnpm_bundle_path` was only ever called from this path and is
  now dead — deleted. `install_command` no longer needs `&self` since none
  of its three branches read anything off the receiver; it's a free function
  on `WorkEnv` now.

Also tightens the `validate_revision_list` comment in `verify.rs`. The
previous wording cited `HEAD@{1}` as a reason for allowing `@`, but `{`
and `}` are still rejected — the comment was misleading. Reflog
revisions aren't supported (and shouldn't be: `{` is a shell
metacharacter we don't want in bench-dir names). The real reason `@` is
allowed is `v1.0.0@sha`-style tag suffixes.

Verified `cargo clippy --locked --workspace --all-targets -- -D warnings`,
`cargo fmt --all -- --check`, and `cargo test -p pacquet-integrated-benchmark`
(5/5 pass) clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
pacquet/tasks/integrated-benchmark/src/work_env.rs (1)

117-122: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

The --work-env path is still shell-exposed.

This avoids embedding --work-env inside install.bash, but hyperfine still runs bash {script_path} with the absolute script path unquoted. A work env containing spaces or shell metacharacters will still break before the script executes.

🔧 Proposed fix
 fn bash_command(&self, id: BenchId) -> String {
     let script_path = self.script_path(id);
-    let script_path = script_path.to_str().expect("convert script path to UTF-8");
+    let script_path = script_path.maybe_quote().to_string();
     format!("bash {script_path}")
 }

Also applies to: 161-161

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pacquet/tasks/integrated-benchmark/src/work_env.rs` around lines 117 - 122,
The absolute script path passed to hyperfine is unquoted, so a malicious or
space-containing --work-env breaks execution; locate where hyperfine is invoked
(the code that builds the command string referencing install.bash / script_path,
e.g., the function building the hyperfine command) and ensure the script path is
safely quoted/escaped before inserting into the shell command (or, better, use a
proper shell-escaping helper such as shlex::quote or build an argv array instead
of a single unescaped string). Apply the same fix to the second occurrence noted
(around the other reference at the same file). Ensure any single quotes in the
path are escaped properly if you use single-quote wrapping.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@pacquet/tasks/integrated-benchmark/src/work_env.rs`:
- Around line 117-122: The absolute script path passed to hyperfine is unquoted,
so a malicious or space-containing --work-env breaks execution; locate where
hyperfine is invoked (the code that builds the command string referencing
install.bash / script_path, e.g., the function building the hyperfine command)
and ensure the script path is safely quoted/escaped before inserting into the
shell command (or, better, use a proper shell-escaping helper such as
shlex::quote or build an argv array instead of a single unescaped string). Apply
the same fix to the second occurrence noted (around the other reference at the
same file). Ensure any single quotes in the path are escaped properly if you use
single-quote wrapping.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 587944c4-687f-48d1-9a10-ca751c7a64c7

📥 Commits

Reviewing files that changed from the base of the PR and between 273e4e3 and 2a886bb.

📒 Files selected for processing (2)
  • pacquet/tasks/integrated-benchmark/src/verify.rs
  • pacquet/tasks/integrated-benchmark/src/work_env.rs
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: Lint and Test (macos-latest)
  • GitHub Check: Lint and Test (ubuntu-latest)
  • GitHub Check: Lint and Test (windows-latest)
  • GitHub Check: Code Coverage
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Compile & Lint
🧰 Additional context used
📓 Path-based instructions (1)
pacquet/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

pacquet/**/*.rs: When porting a function that fires pnpm:<channel> events through globalLogger, logger.debug(), or streamParser.write(), mirror the call site, payload, and ordering so the reporter parses pacquet's NDJSON the same way it parses pnpm's.
Declare a newtype wrapper for branded string types. Do not collapse the brand into a plain String or &str.
If upstream always validates before construction, validate in pacquet's wrapper too. The wrapper must construct only via TryFrom<String> and/or FromStr. Do not provide an infallible public constructor.
If upstream never validates, just brand for type-safety. Expose an infallible From<String> (and From<&str> when convenient).
If upstream occasionally constructs without validation, expose from_str_unchecked as an escape hatch alongside the validating constructor.
Match upstream serde behavior for branded types that cross JSON, YAML, or INI boundaries. Use #[serde(try_from = "String")] for deserialization and #[serde(into = "String")] for serialization.
Use #[derive(derive_more::From)] and #[derive(derive_more::Into)] for mechanical conversion impls. Fall back to manual impl only when conversion needs custom logic.
String-literal unions should become enums, not newtype wrappers. Model closed sets of valid string values as enums.
Template literal types should be treated as branded strings with validation discipline from rules 2-5.
Choose owned vs. borrowed parameters to minimize copies. Widen to the most encompassing type (&Path over &PathBuf, &str over &String) when it doesn't force extra copies.
Prefer Arc::clone(&x) / Rc::clone(&x) over x.clone() for reference-counted types, so the cost is visible at the call site.
Follow Rust API Guidelines for naming conventions.
Do not use star imports inside module bodies. Write use super::{Foo, bar} instead of use super::*;. Two forms stay allowed: external-crate preludes like use rayon::prelude::*; and root-of-module re-...

Files:

  • pacquet/tasks/integrated-benchmark/src/verify.rs
  • pacquet/tasks/integrated-benchmark/src/work_env.rs
🧠 Learnings (1)
📚 Learning: 2026-05-20T23:07:58.444Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11784
File: pacquet/crates/resolving-deps-resolver/src/hoist_peers.rs:120-133
Timestamp: 2026-05-20T23:07:58.444Z
Learning: When reviewing code in this pacquet Rust port, follow the upstream pnpm compatibility rule: only match pnpm’s behavior exactly. Do not propose review changes that intentionally deviate from pnpm’s documented/observed behavior, even if pnpm appears buggy. If you identify a real bug in pnpm behavior, the review should prioritize fixing it upstream in pnpm first, and avoid implementing a pnpm-behavior workaround here unless the same fix has already landed upstream.

Applied to files:

  • pacquet/tasks/integrated-benchmark/src/verify.rs
  • pacquet/tasks/integrated-benchmark/src/work_env.rs
🔇 Additional comments (1)
pacquet/tasks/integrated-benchmark/src/verify.rs (1)

15-18: ⚡ Quick win

Unable to rewrite review comment: no original review text (the content inside <review_comment>...</review_comment>) was provided, so there’s nothing to update.

…s-warm

zkochan noted on PR #11741 that pacquet now implements all three of the
"pnpm-only" scenarios, so the startup rejection in main.rs is no longer
warranted. Confirmed on origin/main:

- **peek (`pacquet add <pkg>`)** — `cli_args/add.rs` exposes `Add` with
  full `--save-prod/dev/optional/peer` flag set; `pacquet_package_manager::Add`
  is the install pipeline behind it.
- **full-resolution (`pacquet install` without `--frozen-lockfile`)** —
  `install.rs:594-623` dispatches to `InstallWithoutLockfile::run`,
  which runs the full resolver chain (`NpmResolver` →
  `GitResolver` → `TarballResolver` → local / runtime / named-registry
  resolvers) against the manifest, downloading and installing real
  packages. Gated only by `config.lockfile = false` (which the bench
  fixture sets via `lockfile=false` in `.npmrc` for the FullResolution
  scenario).
- **gvs-warm (`pacquet install --frozen-lockfile` + `enableGlobalVirtualStore:
  true`)** — `install_frozen_lockfile.rs:540` and `install.rs:556-568`
  read `config.enable_global_virtual_store` and route slot layout
  through the GVS shape (`virtual_store_layout.rs:170-175`). The
  bench's workspace-manifest synthesizer already flips
  `enable_global_virtual_store: true` for `GvsWarm`.

Changes:

- Drop `BenchmarkScenario::supports_pacquet()` — every scenario now
  supports both kinds. The method was only called from one site.
- Drop the startup panic in `main.rs` that rejected pacquet targets
  for the three scenarios above.
- Drop the "Pnpm-only" suffix from the three rustdoc lines on
  `BenchmarkScenario::{Peek, FullResolution, GvsWarm}` — the CLI
  `--help` no longer claims a restriction that doesn't exist.

Verified `cargo clippy --locked --workspace --all-targets -- -D warnings`,
`cargo fmt --all -- --check`, and `cargo test -p pacquet-integrated-benchmark`
(5/5 pass) clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
Copilot AI review requested due to automatic review settings May 21, 2026 06:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 4 comments.

Comment on lines 77 to 81
fn bash_command(&self, id: BenchId) -> String {
let script_path = self.script_path(id);
let script_path = script_path.to_str().expect("convert script path to UTF-8");
format!("bash {script_path}")
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — fixed in 91712f4.

bash_command() now wraps the script path in os_display::Quotable::maybe_quote():

format!("bash {}", self.script_path(id).maybe_quote())

maybe_quote() is a no-op for the common alphanumeric/slash/dash path and only adds shell-safe quoting when the path contains a metacharacter, so the common case stays readable in CI logs while paths with spaces / metacharacters from a user-supplied --work-env are now safe.


Generated by Claude Code

Comment on lines +234 to +236
.arg("run")
.arg("compile")
.pipe(executor("pnpm run compile"));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — confirmed and fixed in 91712f4.

Verified the script chain:

  • Root compile = pn compile-only && pn update-manifests
  • update-manifests = pn meta-updater && pn install

So yes, every bench-revision build was triggering an extra full pnpm install plus a meta-updater pass that rewrites tracked manifests — pure overhead for the bench, and the rewrite-on-second-run was also part of why sync_bench_repo needs its git reset --hard guard.

Went with pnpm run compile-only rather than pnpm --filter=pnpm compile. compile-only is tsgo --build workspace/workspace-manifest-reader workspace/projects-reader && typecheck-only.ts && pn -F=pnpm compile — keeps the workspace-manifest-reader / typecheck-only setup steps that pn -F=pnpm compile depends on, and only drops update-manifests. The pn -F=pnpm compile at the end is what actually produces pnpm/dist/pnpm.{mjs,cjs}, which is what the install script reads. The narrower --filter=pnpm compile would skip the setup, which works in the steady state (tsgo --build cascades through TS project references) but is more likely to break in edge cases (fresh worktree, post-rebase, etc.); compile-only is the documented entry point for "build without rewriting the repo" so it's the safer level.


Generated by Claude Code

Comment on lines 18 to +24
/// Path to the git repository of pacquet.
#[clap(long, short = 'R', default_value = ".")]
pub repository: PathBuf,

/// Path to pnpm's git repository. Required when a target is `pnpm@<rev>`. Defaults to `--repository`.
#[clap(long)]
pub pnpm_repository: Option<PathBuf>,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — help text fixed in 91712f4.

You're right that the "Required when a target is pnpm@<rev>" wording was inaccurate: the orchestrator falls back to --repository, and for the common pnpm/pnpm monorepo case that's exactly the right answer (both pacquet's source and pnpm's source live in the same tree). The flag is only contextually required — needed iff --repository isn't a pnpm checkout.

Rewrote the help to say so:

/// Path to pnpm's git repository. Only set this if pnpm and pacquet
/// live in separate clones; defaults to `--repository`, which is
/// correct for the `pnpm/pnpm` monorepo (where both live together).

Didn't add a clap-level constraint. requires or required_if_eq_any can't express the rule (the predicate is "--repository doesn't look like a pnpm checkout", which clap can't evaluate). The runtime path in main.rs calls ensure_pnpm_git_repo(pnpm_repository.unwrap_or(&repository)) which already produces an actionable panic message ("X doesn't look like a pnpm checkout — expected pnpm/package.json or pnpm-workspace.yaml at the root") when the fallback fails. That covers the diagnostic gap.


Generated by Claude Code

Comment on lines +28 to +40
/// Assert that `path` is a git checkout of the pnpm codebase. Recognizes both
/// a standalone pnpm clone (top-level `pnpm/package.json`) and the
/// pnpm-as-monorepo layout this repo uses (a `pnpm-workspace.yaml` at the
/// root), so the same orchestrator binary works for both shapes.
pub fn ensure_pnpm_git_repo(path: &Path) {
ensure_git_repo_common(path);
let has_pnpm_dir = path.join("pnpm").join("package.json").is_file();
let has_workspace_yaml = path.join("pnpm-workspace.yaml").is_file();
assert!(
has_pnpm_dir || has_workspace_yaml,
"{path:?} doesn't look like a pnpm checkout — \
expected `pnpm/package.json` or `pnpm-workspace.yaml` at the root",
);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — docstring fixed in 91712f4.

You're right that the "standalone pnpm clone (top-level pnpm/package.json)" phrasing was misleading: pnpm doesn't ship a layout where its source lives at the repo root — the pnpm/pnpm repo IS the monorepo, with pnpm/package.json as the CLI package's manifest inside it. The function checks for that file OR for pnpm-workspace.yaml at the root; either marker is enough. The "standalone" word was doing all the work in the wrong direction.

New docstring:

/// Assert that `path` is a git checkout of pnpm's source — the
/// `pnpm/pnpm` monorepo. Looks for `pnpm/package.json` (the CLI
/// package's manifest, present in every revision since pnpm became a
/// monorepo) or `pnpm-workspace.yaml` at the root; either marker is
/// enough. Doesn't support a hypothetical layout where pnpm's source
/// lives directly at the repo root (no `pnpm/` subdir, no workspace
/// manifest) — that's not a shape pnpm ships in.

Validation behavior is unchanged.


Generated by Claude Code

Four follow-up findings from Copilot, all valid:

- **bash_command quoting (work_env.rs:77)** — hyperfine runs each
  benchmarked command through a shell, so the `bash {path}` string
  must survive shell-tokenization. `path` is `<work_env>/<bench_id>
  /install.bash` and `--work-env` is user-supplied, so a path with
  spaces / metacharacters would break the run. Wrap the script path
  in `os_display::Quotable::maybe_quote()` so the result is
  shell-safe (and bare for the common alphanumeric/slash/dash case).

- **`pnpm run compile` is too heavy (work_env.rs:236)** — the root
  `compile` script is `pn compile-only && pn update-manifests`, and
  `update-manifests` fires an extra `pnpm install` plus `pn
  meta-updater` (which rewrites tracked manifest files). The bench
  doesn't need any of that — it only needs `pnpm/dist/pnpm.{mjs,cjs}`,
  which `compile-only` already produces via `tsgo --build` →
  `typecheck-only` → `pn -F=pnpm compile`. Switching to
  `compile-only` drops the redundant install on every revision build.
  As a side benefit the second-invocation `git reset --hard` in
  `sync_bench_repo` will have less to discard (lockfile / generated
  metadata won't drift). Comment references that mentioned
  `pnpm run compile` updated to track the new name.

- **`--pnpm-repository` help text was misleading (cli_args.rs:22-24)** —
  the old "Required when a target is `pnpm@<rev>`" was inaccurate
  because the orchestrator falls back to `--repository`, which is
  exactly what you want for the pnpm/pnpm monorepo case (both
  pacquet and pnpm live in the same repo). Rewrote as: "Only set
  this if pnpm and pacquet live in separate clones; defaults to
  `--repository`, which is correct for the `pnpm/pnpm` monorepo."
  No clap-level constraint added — the contextual rule ("required
  iff `--repository` isn't a pnpm checkout") can't be expressed in
  `required_if_eq` / `requires`, and `ensure_pnpm_git_repo` already
  panics with an actionable message when the fallback isn't a pnpm
  checkout.

- **`ensure_pnpm_git_repo` docstring (verify.rs:31-37)** — the
  "standalone pnpm clone" phrasing was confusing: pnpm doesn't ship
  a layout where its source lives at the repo root, so "standalone"
  is misleading. The function actually recognizes the pnpm/pnpm
  monorepo via either `pnpm/package.json` (CLI manifest) or
  top-level `pnpm-workspace.yaml`. Reworded to say so explicitly
  and to call out that the no-`pnpm/`-subdir shape isn't supported
  because pnpm doesn't have one.

Verified `cargo clippy --locked --workspace --all-targets -- -D warnings`,
`cargo fmt --all -- --check`, and `cargo test -p pacquet-integrated-benchmark`
(5/5 pass) clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
load_lockfile().into().pipe(Some)
}
}
if self.lockfile_enabled() { Some(load_lockfile().into()) } else { None }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code,

What do you think about this refactor?

Suggested change
if self.lockfile_enabled() { Some(load_lockfile().into()) } else { None }
self.lockfile_enabled()
.then_some(())
.map(|()| load_lockfile())
.map(String::from)

If it's good, apply.

If the idea is good, but the code is incorrect, fix it and apply.

If it's not good, explain.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idea is good but the suggested code doesn't compile — applied a corrected variant. Details below.

The compile failure. .map(String::from) requires String: From<T>, which Rust's trait solver doesn't infer from T: Into<String> even though the blanket makes them equivalent. Verified by typing it out:

fn lockfile<T: Into<String>, F: FnOnce() -> T>(en: bool, f: F) -> Option<String> {
    en.then_some(()).map(|()| f()).map(String::from)
}
error[E0277]: the trait bound `String: From<T>` is not satisfied
   |
6 |     en.then_some(()).map(|()| f()).map(String::from)
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
help: consider extending the `where` clause, but there might be an alternative better way to express this requirement
   |
   |     F: FnOnce() -> T, String: From<T>
   |                       +++++++++++++++

Two fixes that keep the existing bound intact:

// Option A — your chain with .map(Into::into) instead of String::from:
en.then_some(()).map(|()| f()).map(Into::into)

// Option B — collapse to Option::then, which already gates a closure on a bool:
en.then(|| f().into())

Both compile. Applied Option B in edbe696:

self.lockfile_enabled().then(|| load_lockfile().into())

Why B over A: Option::then is the idiomatic Rust spelling of "produce Some(value) iff bool — run the closure to get the value", which is exactly what we want. The chain in A goes through an extra .then_some(()) + .map(|()| …) step whose only job is to drag the eventual value through a unit-typed Option<()> — readable enough but two more steps and the |()| destructure pattern is uncommon. B keeps the original's single-expression shape, just rewritten functionally.

Verified cargo clippy --locked --workspace --all-targets -- -D warnings, cargo fmt --all -- --check, and cargo test -p pacquet-integrated-benchmark (5/5 pass) clean.


Generated by Claude Code

KSX suggested rewriting the imperative `if … { Some(…) } else { None }`
as a functional chain. The exact suggested form
(`.then_some(()).map(|()| f()).map(String::from)`) doesn't compile under
the existing `Text: Into<String>` bound — `String::from(t)` requires the
inverse `String: From<T>`, which Rust's trait solver doesn't infer from
`T: Into<String>` despite the blanket. The fix that keeps the bound
intact is `.map(Into::into)` (or, more compactly, fold the whole
gate-on-bool into `Option::then`):

    self.lockfile_enabled().then(|| load_lockfile().into())

That's the idiomatic Rust for "produce `Some(value)` iff bool", drops
the `.then_some(()).map(|()| …)` two-step destructure pattern that's
unusual, and lines up with the original's single-expression shape.

Verified `cargo clippy --locked --workspace --all-targets -- -D warnings`,
`cargo fmt --all -- --check`, and `cargo test -p pacquet-integrated-benchmark`
(5/5 pass) clean.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
Copilot AI review requested due to automatic review settings May 21, 2026 07:50

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pacquet/tasks/integrated-benchmark/src/cli_args.rs (1)

253-253: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fix hyperfine flag typo (--ignore-failure not --ignore-failures) (pacquet/tasks/integrated-benchmark/src/cli_args.rs:253)

Hyperfine’s CLI uses --ignore-failure (singular); --ignore-failures will be treated as an unrecognized flag.

🐛 Proposed fix
         if ignore_failure {
-            hyperfine_command.arg("--ignore-failures");
+            hyperfine_command.arg("--ignore-failure");
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pacquet/tasks/integrated-benchmark/src/cli_args.rs` at line 253, The CLI
currently passes the incorrect hyperfine flag "--ignore-failures"; update the
call that builds the hyperfine command (the hyperfine_command.arg(...)
invocation) to use the correct singular flag "--ignore-failure" so hyperfine
recognizes it; locate the hyperfine_command construction and replace the
argument string accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@pacquet/tasks/integrated-benchmark/src/cli_args.rs`:
- Line 253: The CLI currently passes the incorrect hyperfine flag
"--ignore-failures"; update the call that builds the hyperfine command (the
hyperfine_command.arg(...) invocation) to use the correct singular flag
"--ignore-failure" so hyperfine recognizes it; locate the hyperfine_command
construction and replace the argument string accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 91d53554-00e7-439e-988a-1dc977fa4a5e

📥 Commits

Reviewing files that changed from the base of the PR and between 91712f4 and edbe696.

📒 Files selected for processing (1)
  • pacquet/tasks/integrated-benchmark/src/cli_args.rs
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
  • GitHub Check: Agent
  • GitHub Check: Code Coverage
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Lint and Test (ubuntu-latest)
  • GitHub Check: Run benchmark on ubuntu-latest
  • GitHub Check: Lint and Test (macos-latest)
  • GitHub Check: Lint and Test (windows-latest)
  • GitHub Check: Dylint
  • GitHub Check: Compile & Lint
🧰 Additional context used
📓 Path-based instructions (1)
pacquet/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

pacquet/**/*.rs: When porting a function that fires pnpm:<channel> events through globalLogger, logger.debug(), or streamParser.write(), mirror the call site, payload, and ordering so the reporter parses pacquet's NDJSON the same way it parses pnpm's.
Declare a newtype wrapper for branded string types. Do not collapse the brand into a plain String or &str.
If upstream always validates before construction, validate in pacquet's wrapper too. The wrapper must construct only via TryFrom<String> and/or FromStr. Do not provide an infallible public constructor.
If upstream never validates, just brand for type-safety. Expose an infallible From<String> (and From<&str> when convenient).
If upstream occasionally constructs without validation, expose from_str_unchecked as an escape hatch alongside the validating constructor.
Match upstream serde behavior for branded types that cross JSON, YAML, or INI boundaries. Use #[serde(try_from = "String")] for deserialization and #[serde(into = "String")] for serialization.
Use #[derive(derive_more::From)] and #[derive(derive_more::Into)] for mechanical conversion impls. Fall back to manual impl only when conversion needs custom logic.
String-literal unions should become enums, not newtype wrappers. Model closed sets of valid string values as enums.
Template literal types should be treated as branded strings with validation discipline from rules 2-5.
Choose owned vs. borrowed parameters to minimize copies. Widen to the most encompassing type (&Path over &PathBuf, &str over &String) when it doesn't force extra copies.
Prefer Arc::clone(&x) / Rc::clone(&x) over x.clone() for reference-counted types, so the cost is visible at the call site.
Follow Rust API Guidelines for naming conventions.
Do not use star imports inside module bodies. Write use super::{Foo, bar} instead of use super::*;. Two forms stay allowed: external-crate preludes like use rayon::prelude::*; and root-of-module re-...

Files:

  • pacquet/tasks/integrated-benchmark/src/cli_args.rs
🧠 Learnings (1)
📚 Learning: 2026-05-20T23:07:58.444Z
Learnt from: zkochan
Repo: pnpm/pnpm PR: 11784
File: pacquet/crates/resolving-deps-resolver/src/hoist_peers.rs:120-133
Timestamp: 2026-05-20T23:07:58.444Z
Learning: When reviewing code in this pacquet Rust port, follow the upstream pnpm compatibility rule: only match pnpm’s behavior exactly. Do not propose review changes that intentionally deviate from pnpm’s documented/observed behavior, even if pnpm appears buggy. If you identify a real bug in pnpm behavior, the review should prioritize fixing it upstream in pnpm first, and avoid implementing a pnpm-behavior workaround here unless the same fix has already landed upstream.

Applied to files:

  • pacquet/tasks/integrated-benchmark/src/cli_args.rs
🔇 Additional comments (1)
pacquet/tasks/integrated-benchmark/src/cli_args.rs (1)

167-167: LGTM!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 3 comments.

let mut command = format!("rm -rf {remove_targets}");

for dir in &dirs {
for (dst, src) in cleanup.restore {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stupid fucking Copilot. Stop replacing the compiler. The compiler knows better than you. And if the compiler agrees, who are you to disagree?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incorrect — the code compiles and binds the right types. Skipping.

cleanup.restore has type &'static [(&'static str, &'static str)]. The expression for (dst, src) in cleanup.restore does work via Rust's match ergonomics (also known as binding modes): when a pattern without a leading & is matched against a &T, the bindings inside automatically become references. So (dst, src) matched against &(&str, &str) gives dst: &&str and src: &&str, which PathBuf::join accepts via AsRef<Path> and deref coercion.

Reduced verification:

fn main() {
    let xs: &[(&str, &str)] = &[("a", "b"), ("c", "d")];
    for (dst, src) in xs {
        let _: &&str = dst;
        let _: &&str = src;
        println!("{src} -> {dst}");
    }
}
$ rustc --edition=2024 test.rs && ./test
b -> a
d -> c

The current branch has also been passing cargo clippy --locked --workspace --all-targets -- -D warnings and cargo nextest run on every commit through CI (Lint and Test ubuntu / macos / windows all green on edbe696), which wouldn't be possible if the file failed to compile. The suggested &(dst, src) or .iter().copied() rewrites are unnecessary noise — match ergonomics is idiomatic and the existing form is shorter.

References:


Generated by Claude Code

Comment thread benchmarks/README.md Outdated
```
1. Builds the `integrated-benchmark` binary in release mode.
2. Clones the current repo into the temp work-env once per revision
(`HEAD` and `main`) and runs `pnpm install && pnpm run compile` in each

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — fixed in 004940a. The README was stale after 91712f4 switched the per-revision build step from pnpm run compile to pnpm run compile-only. Updated the step description and added a one-sentence note explaining why compile-only (not the full compile) is the right level.


Generated by Claude Code

Comment thread benchmarks/bench.sh

# Pre-build both revisions once. Subsequent scenario invocations still
# re-enter the orchestrator's build step (sync_bench_repo + pnpm install
# + pnpm run compile), but `pnpm install` is a no-op on the populated
Followup to 91712f4, which changed the per-revision build step from
`pnpm run compile` to `pnpm run compile-only` (to drop the redundant
`update-manifests` pass). The README still said `pnpm run compile`,
which Copilot flagged on PR #11741. Update the step description and
add a one-sentence note explaining why `compile-only` (not the full
`compile`) is the right level.

https://claude.ai/code/session_01Ya2xTvnu5ycwqux8tfPbn5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants