ci(bencher): record benchmark results to Bencher by zkochan · Pull Request #11875 · pnpm/pnpm

zkochan · 2026-05-23T10:26:35Z

Summary

Wires both benchmark workflows to Bencher so install-perf history is tracked continuously instead of only being visible in per-PR comments. Both stacks live in one Bencher project (slug: pnpm) under separate testbeds (pnpm, pacquet) so the scenario charts are directly comparable.

benchmarks/bench.sh emits a hyperfine-shaped bencher-results.json that combines the six pnpm scenarios — @HEAD result only, command renamed to the scenario name so Bencher uses the scenario as the benchmark identifier.
.github/workflows/benchmark.yml (pnpm TS bench): adds push: branches: [main] so each merge updates the pnpm baseline, then uploads via bencher run --testbed pnpm. Branch policy: main on push, pr/<n> on manual PR dispatch, ref_name otherwise (all forked from main).
.github/workflows/pacquet-integrated-benchmark.yml: adds the same main-push baseline for the pacquet testbed, builds the Bencher report from the four scenario JSONs, and stages it into the existing artifact. Inline bencher run only fires on push to main (where secrets are available).
.github/workflows/pacquet-integrated-benchmark-comment.yml: after the existing summary comment, downloads bencher-results.json from the artifact and runs bencher run --branch pr/<n> --start-point main --start-point-reset --ci-number <n>. Runs in the trusted workflow_run privilege context, so fork PRs are covered too.

The existing hand-rolled PR comments are kept alongside Bencher's auto-comment during rollout; we can drop them once thresholds are tuned.

Required out-of-band setup

Create a Bencher project with slug pnpm at bencher.dev. Testbeds (pnpm, pacquet), the main branch, and scenario benchmarks will auto-create on the first push.
Add BENCHER_API_TOKEN to repo secrets (Actions). Without it, the new steps no-op with a ::notice:: — so this PR is safe to merge before the token lands, but Bencher won't actually record anything until both are in place.

Out of scope

pacquet-micro-benchmark.yml is unchanged. It uses criterion, not hyperfine, and would need Bencher's rust_criterion adapter. Easy follow-up if useful.
No changeset added — these are CI/script changes, no published package is touched.

Test plan

Merge → first main push runs benchmark.yml and pacquet-integrated-benchmark.yml; both should attempt their Bencher upload steps. With the token unset they should emit ::notice::BENCHER_API_TOKEN not set and exit 0.
After creating the Bencher project and adding the secret, re-run benchmark.yml manually with a recent PR number and verify Bencher comments on the PR.
Trigger the pacquet benchmark on a PR (changes under pacquet/**) and confirm both the existing summary comment and Bencher's comment appear.
Confirm fork PRs flow correctly — the inline upload step is gated to github.event_name == 'push', so fork PRs skip it and rely entirely on the comment workflow.

Written by an agent (Claude Code, claude-opus-4-7).

Summary by CodeRabbit

New Features
- CI now can install a Bencher client and automatically upload consolidated Bencher-compatible benchmark results for PRs and main-branch pushes when configured; uploads are skipped if credentials or results are absent.
- Bench runner produces a single bencher-results.json and CI stages/uploads it; local tooling can optionally generate this report when JSON tooling is present.
Documentation
- Docs and contributor guidance updated with revised scenario names, new scenario descriptions, and updated example benchmark commands.

Tracks both stacks in one Bencher project (`pnpm`), under separate testbeds (`pnpm`, `pacquet`). - `benchmarks/bench.sh` emits a hyperfine-shaped `bencher-results.json` combining the six pnpm scenarios. - `benchmark.yml` adds `push: branches: [main]` so each merge updates the `pnpm` baseline, then uploads via the Bencher CLI. - `pacquet-integrated-benchmark.yml` adds the same main-push baseline for `pacquet`, combines the four scenario JSONs into a Bencher report, and stages it into the existing artifact. - `pacquet-integrated-benchmark-comment.yml` uploads PR results from the trusted `workflow_run` context so fork PRs are covered too. Requires `BENCHER_API_TOKEN` in repo secrets; workflows no-op with a `::notice::` if it's missing.

coderabbitai · 2026-05-23T10:26:42Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 625a48a8-047d-4884-8f7d-7c003a2bbaf9

📥 Commits

Reviewing files that changed from the base of the PR and between d1e2f0b and 9471306.

📒 Files selected for processing (1)

pacquet/tasks/integrated-benchmark/src/cli_args.rs

📜 Recent review details

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)

GitHub Check: Lint and Test (windows-latest)
GitHub Check: Lint and Test (ubuntu-latest)
GitHub Check: Code Coverage
GitHub Check: Run benchmark on ubuntu-latest
GitHub Check: Run benchmark on ubuntu-latest

🧰 Additional context used

📓 Path-based instructions (1)

pacquet/**/*.rs

📄 CodeRabbit inference engine (pacquet/AGENTS.md)

pacquet/**/*.rs: When porting a function that fires pnpm:<channel> events through globalLogger, logger.debug(), or streamParser.write(), mirror the call site, payload, and ordering so the reporter parses pacquet's NDJSON the same way it parses pnpm's.
Declare a newtype wrapper for branded string types. Do not collapse the brand into a plain String or &str.
If upstream always validates before construction, validate in pacquet's wrapper too. The wrapper must construct only via TryFrom<String> and/or FromStr. Do not provide an infallible public constructor.
If upstream never validates, just brand for type-safety. Expose an infallible From<String> (and From<&str> when convenient).
If upstream occasionally constructs without validation, expose from_str_unchecked as an escape hatch alongside the validating constructor.
Match upstream serde behavior for branded types that cross JSON, YAML, or INI boundaries. Use #[serde(try_from = "String")] for deserialization and #[serde(into = "String")] for serialization.
Use #[derive(derive_more::From)] and #[derive(derive_more::Into)] for mechanical conversion impls. Fall back to manual impl only when conversion needs custom logic.
String-literal unions should become enums, not newtype wrappers. Model closed sets of valid string values as enums.
Template literal types should be treated as branded strings with validation discipline from rules 2-5.
Choose owned vs. borrowed parameters to minimize copies. Widen to the most encompassing type (&Path over &PathBuf, &str over &String) when it doesn't force extra copies.
Prefer Arc::clone(&x) / Rc::clone(&x) over x.clone() for reference-counted types, so the cost is visible at the call site.
Follow Rust API Guidelines for naming conventions.
Do not use star imports inside module bodies. Write use super::{Foo, bar} instead of use super::*;. Two forms stay allowed: external-crate preludes like use rayon::prelude::*; and root-of-module re-...

Files:

pacquet/tasks/integrated-benchmark/src/cli_args.rs

🧠 Learnings (1)

📚 Learning: 2026-05-20T23:07:58.444Z

Learnt from: zkochan
Repo: pnpm/pnpm PR: 11784
File: pacquet/crates/resolving-deps-resolver/src/hoist_peers.rs:120-133
Timestamp: 2026-05-20T23:07:58.444Z
Learning: When reviewing code in this pacquet Rust port, follow the upstream pnpm compatibility rule: only match pnpm’s behavior exactly. Do not propose review changes that intentionally deviate from pnpm’s documented/observed behavior, even if pnpm appears buggy. If you identify a real bug in pnpm behavior, the review should prioritize fixing it upstream in pnpm first, and avoid implementing a pnpm-behavior workaround here unless the same fix has already landed upstream.

Applied to files:

pacquet/tasks/integrated-benchmark/src/cli_args.rs

🔇 Additional comments (1)

pacquet/tasks/integrated-benchmark/src/cli_args.rs (1)

155-157: LGTM!

📝 Walkthrough

Walkthrough

Adds generation of a Bencher-compatible bencher-results.json, documents it, and updates three GitHub Actions workflows to trigger on push, install the Bencher CLI, and upload results to Bencher with conditional token gating and branch-aware configuration.

Changes

Bencher Integration for Continuous Benchmark Tracking

Layer / File(s)	Summary
Bencher results generation in benchmark script `benchmarks/bench.sh`, `benchmarks/README.md`, `pacquet/CONTRIBUTING.md`	Converts per-scenario hyperfine JSONs to a single `bencher-results.json` by selecting `@HEAD` results and renaming `command` to the scenario name; documents the new artifact and updates example commands.
Pacquet integrated benchmark workflow `.github/workflows/pacquet-integrated-benchmark.yml`	Adds `push`->`main` trigger (with paths), runs new `fresh-*` scenarios, builds `bencher-results.json` with jq, stages `SUMMARY.md` and `bencher-results.json`, installs Bencher CLI, and uploads results with main-aware start-point logic.
Generic benchmark workflow Bencher integration `.github/workflows/benchmark.yml`	Adds `push`->`main` trigger, conditionally installs Bencher CLI when benchmark output exists, validates `BENCHER_API_TOKEN` and `bencher-results.json`, then runs `bencher run` with branch/start-point selection based on event/ref/inputs.
PR comment workflow Bencher upload `.github/workflows/pacquet-integrated-benchmark-comment.yml`	Adds conditional Bencher CLI install and upload step that uses `benchmark-artifact/bencher-results.json`, exits early if `BENCHER_API_TOKEN` is unset, and runs `bencher run` configured for the PR branch with CI metadata.
Benchmark CLI scenario model update `pacquet/tasks/integrated-benchmark/src/cli_args.rs`, `pacquet/tasks/integrated-benchmark/src/work_env.rs`	Replaces `BenchmarkScenario` enum with new `Fresh`/`Restore`/`AddDep*` variants, updates install args, lockfile and cleanup behavior per scenario, GVS predicate, and clarifies per-iteration restore behavior in comments.

Sequence Diagram(s)

sequenceDiagram
  participant BenchScript as benchmarks/bench.sh
  participant PacquetWF as .github/workflows/pacquet-integrated-benchmark.yml
  participant ArtifactStore as benchmark-artifact
  participant BencherCLI as bencherdev/bencher
  participant Bencher as Bencher

  BenchScript->>PacquetWF: produce per-scenario BENCHMARK_REPORT*.json
  PacquetWF->>PacquetWF: transform (jq) -> bencher-results.json
  PacquetWF->>ArtifactStore: stage SUMMARY.md + bencher-results.json
  PacquetWF->>BencherCLI: install (if artifact present)
  BencherCLI->>Bencher: bencher run --file bench-work-env/bencher-results.json

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

pnpm/pnpm#11741: Both PRs modify the integrated-benchmark Rust code that defines BenchmarkScenario/scenario-related CLI behavior.
pnpm/pnpm#11838: Modifies pacquet/tasks/integrated-benchmark/src/cli_args.rs with related scenario/install-args changes.
pnpm/pnpm#11643: Overlaps with workflow paths filtering and benchmark workflow configuration changes.

Suggested reviewers

KSXGitHub

Poem

🐰 I hopped through JSON, jq, and CI light,
Merged HEAD's numbers into one tidy sight;
Workflows now whisper to the Bencher sky,
Tokens snug, branches waving hi—
Metrics hop onward, tracking each small fight.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: wiring benchmark workflows to record results to Bencher, which is the primary objective of this PR.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch bencher

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-05-23T10:51:56Z

Integrated-Benchmark Report (Linux)

Scenario: Isolated linker: fresh restore, cold cache + cold store

Command	Mean [s]	Min [s]	Max [s]	Relative
`pacquet@HEAD`	2.456 ± 0.093	2.335	2.601	1.02 ± 0.05
`pacquet@main`	2.409 ± 0.056	2.328	2.501	1.00
`pnpm`	4.987 ± 0.059	4.898	5.062	2.07 ± 0.05

BENCHMARK_REPORT.json

{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 2.45578310486,
      "stddev": 0.09326222936858554,
      "median": 2.4691350727600003,
      "user": 2.72747364,
      "system": 3.64591852,
      "min": 2.3349557407600003,
      "max": 2.60107321976,
      "times": [
        2.58224209276,
        2.3777718727600003,
        2.5043210297600003,
        2.4853639977600004,
        2.60107321976,
        2.3904305587600003,
        2.46731795576,
        2.34340239076,
        2.47095218976,
        2.3349557407600003
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 2.40877509376,
      "stddev": 0.055796526661784515,
      "median": 2.39292858376,
      "user": 2.7326028399999993,
      "system": 3.64037572,
      "min": 2.32833523776,
      "max": 2.50067259976,
      "times": [
        2.4387470277600003,
        2.3781797747600004,
        2.32833523776,
        2.40174072876,
        2.50067259976,
        2.4442199487600003,
        2.37892379076,
        2.3518024467600003,
        2.48101294376,
        2.38411643876
      ]
    },
    {
      "command": "pnpm",
      "mean": 4.98704360176,
      "stddev": 0.059346966314228956,
      "median": 4.97784088926,
      "user": 8.512679539999999,
      "system": 4.29128902,
      "min": 4.89777282276,
      "max": 5.0618745587600005,
      "times": [
        5.05509064976,
        4.97504024076,
        4.90549900376,
        4.97434922776,
        5.03395664576,
        5.0618745587600005,
        4.9471211557600006,
        4.98064153776,
        5.03909017476,
        4.89777282276
      ]
    }
  ]
}

Scenario: Isolated linker: fresh restore, hot cache + hot store

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`pacquet@HEAD`	660.6 ± 26.8	641.9	731.4	1.00
`pacquet@main`	687.5 ± 72.7	644.1	884.6	1.04 ± 0.12
`pnpm`	2676.4 ± 141.4	2544.5	3005.5	4.05 ± 0.27

BENCHMARK_REPORT.json

{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 0.66057326798,
      "stddev": 0.026764163366911724,
      "median": 0.65100267388,
      "user": 0.376959,
      "system": 1.4552125200000001,
      "min": 0.64185736538,
      "max": 0.73143110438,
      "times": [
        0.73143110438,
        0.66071642138,
        0.64645952638,
        0.64426809138,
        0.6600363273800001,
        0.64780142738,
        0.67369751838,
        0.64526097738,
        0.65420392038,
        0.64185736538
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 0.6874608495800001,
      "stddev": 0.07271920949142308,
      "median": 0.66199984288,
      "user": 0.3794958,
      "system": 1.4577622200000002,
      "min": 0.64412904038,
      "max": 0.8845756193800001,
      "times": [
        0.70797076638,
        0.67519814038,
        0.64613644638,
        0.69687002338,
        0.65126717638,
        0.64412904038,
        0.65763179238,
        0.8845756193800001,
        0.66636789338,
        0.64446159738
      ]
    },
    {
      "command": "pnpm",
      "mean": 2.67635808238,
      "stddev": 0.14141326031949564,
      "median": 2.62726667838,
      "user": 3.3351839999999995,
      "system": 2.24173212,
      "min": 2.54452487938,
      "max": 3.0054958903799998,
      "times": [
        2.73232054538,
        2.54974553338,
        2.54452487938,
        3.0054958903799998,
        2.59206930038,
        2.65830230738,
        2.59623104938,
        2.59308456538,
        2.79410421238,
        2.69770254038
      ]
    }
  ]
}

Scenario: Isolated linker: fresh install, cold cache + cold store

Command	Mean [s]	Min [s]	Max [s]	Relative
`pacquet@HEAD`	5.004 ± 0.136	4.815	5.192	1.01 ± 0.04
`pacquet@main`	4.967 ± 0.155	4.774	5.232	1.00
`pnpm`	6.894 ± 0.104	6.707	7.022	1.39 ± 0.05

BENCHMARK_REPORT.json

{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 5.00442922534,
      "stddev": 0.13605322345806645,
      "median": 4.980987213840001,
      "user": 6.715868559999999,
      "system": 3.6474317,
      "min": 4.81486797634,
      "max": 5.192435731340001,
      "times": [
        5.17766849834,
        4.95867423334,
        4.89625747134,
        5.043606731340001,
        4.91101108334,
        5.192435731340001,
        4.81486797634,
        5.16560349534,
        5.00330019434,
        4.88086683834
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 4.96670029294,
      "stddev": 0.15456930609280436,
      "median": 4.92934746484,
      "user": 6.7043497599999995,
      "system": 3.6502285,
      "min": 4.77396755034,
      "max": 5.23156078634,
      "times": [
        4.78459658734,
        5.1156350143400005,
        4.89885866034,
        5.23156078634,
        4.77396755034,
        5.164178032340001,
        4.89176830134,
        4.93252076234,
        4.94774306734,
        4.92617416734
      ]
    },
    {
      "command": "pnpm",
      "mean": 6.893589930940001,
      "stddev": 0.10379715535190981,
      "median": 6.90533459134,
      "user": 11.09385476,
      "system": 4.555295499999999,
      "min": 6.70726206434,
      "max": 7.02219315634,
      "times": [
        6.70726206434,
        6.90192451334,
        6.98618751434,
        7.02219315634,
        6.80668045134,
        6.79960062434,
        6.90874466934,
        7.01445588634,
        6.94844889434,
        6.840401535340001
      ]
    }
  ]
}

Scenario: Isolated linker: fresh install, hot cache + hot store

Command	Mean [s]	Min [s]	Max [s]	Relative
`pacquet@HEAD`	3.865 ± 0.089	3.696	3.975	1.02 ± 0.03
`pacquet@main`	3.792 ± 0.067	3.702	3.887	1.00
`pnpm`	4.398 ± 0.052	4.299	4.472	1.16 ± 0.02

BENCHMARK_REPORT.json

{
  "results": [
    {
      "command": "pacquet@HEAD",
      "mean": 3.8645455069000008,
      "stddev": 0.08898497919121567,
      "median": 3.8569782523000002,
      "user": 4.27743982,
      "system": 2.1574152399999997,
      "min": 3.6960852417999996,
      "max": 3.9747887118,
      "times": [
        3.8632898108,
        3.9659548748,
        3.9147713887999998,
        3.8403342228,
        3.9502861528,
        3.7914508678,
        3.8506666938,
        3.6960852417999996,
        3.9747887118,
        3.7978271038
      ]
    },
    {
      "command": "pacquet@main",
      "mean": 3.7919680630999997,
      "stddev": 0.06678050341733613,
      "median": 3.7881918138,
      "user": 4.21596012,
      "system": 2.13757864,
      "min": 3.7022157768,
      "max": 3.8866727817999998,
      "times": [
        3.7957109328,
        3.8374044158,
        3.7022157768,
        3.7404925928,
        3.8466309308,
        3.8866727817999998,
        3.7556964408,
        3.7037095628,
        3.7806726948,
        3.8704745018
      ]
    },
    {
      "command": "pnpm",
      "mean": 4.3982595759,
      "stddev": 0.05226865875660416,
      "median": 4.4051863218000005,
      "user": 5.58554892,
      "system": 2.62874224,
      "min": 4.2989558068,
      "max": 4.472200594799999,
      "times": [
        4.3991476718,
        4.2989558068,
        4.3710678307999995,
        4.4439152048,
        4.4410585798,
        4.4200532868,
        4.472200594799999,
        4.388949580799999,
        4.3360222308,
        4.4112249718
      ]
    }
  ]
}

The inline Bencher upload was gated to `event_name == 'push'`, which meant manual dispatch from a feature branch ran the bench but skipped the upload. Both push and workflow_dispatch execute in the base-repo privilege context, so it's safe to upload from both — the fork-safety gate only needs to keep `pull_request` runs out. On dispatch from a non-main branch we record into the ref name with `--start-point main --start-point-reset`, matching the pnpm bench's branch policy.

qodo-code-review · 2026-05-23T12:10:57Z

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/benchmark.yml:
- Around line 126-131: The branch-selection logic treats workflow_dispatch on
main like a non-main ref; update the conditional so manual dispatches whose
REF_NAME is "main" are handled like push: add an explicit check (e.g., elif [
"$REF_NAME" = "main" ]; then) that sets args+=(--branch main) before the
non-main else branch that currently appends --start-point and
--start-point-reset; reference the existing EVENT_NAME check and the args+=(...)
lines to locate where to insert this new branch.

In `@benchmarks/bench.sh`:
- Around line 128-139: The current if block that builds bencher inputs using jq
silently skips generating bencher-results.json when jq is missing; update the
script around the existing "if command -v jq >/dev/null; then" block so that the
else branch prints a clear error (e.g., "jq is required to produce
bencher-results.json") and exits non‑zero instead of doing nothing. Reference
the same symbols used in the diff (jq, SCENARIOS, bencher_inputs, BENCH_DIR,
bencher-results.json) and ensure the script fails early when jq is not available
so missing tooling is surfaced.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 13ac2fe6-5f83-4b07-a9e7-72c445c80cfe

📥 Commits

Reviewing files that changed from the base of the PR and between 6ae4b5d and 59b2769.

📒 Files selected for processing (5)

.github/workflows/benchmark.yml
.github/workflows/pacquet-integrated-benchmark-comment.yml
.github/workflows/pacquet-integrated-benchmark.yml
benchmarks/README.md
benchmarks/bench.sh

Two small fixes from PR review: - `benchmark.yml`: manual dispatch from `main` was falling into the non-main branch arm, recording `--branch main --start-point main --start-point-reset` — equivalent to forking main from itself. Treat it like a `push` event so a manual run on main updates the baseline directly. Matches the pacquet workflow's branch policy. - `bench.sh`: emit a stderr warning when `jq` is missing instead of silently skipping `bencher-results.json`. Keeps behaviour optional for local users but makes the skip discoverable.

Replaces the hyperfine-leaking names (`clean-install`, `frozen-lockfile`, `peek`, `gvs-warm`, …) with a consistent grid that spells out every state the benchmark depends on: - "Fresh" — node_modules wiped at start (future variants will start with a populated node_modules). - "Install" vs "Restore" vs "Add new dep" — the work being measured. - "hot/cold cache + hot/cold store" — both pnpm directories, spelled out separately because they're distinct on disk. - "isolated linker" — nodeLinker mode (future variants will cover `hoisted` and `pnp`). The slugs map directly from the clap-derived kebab-case names, so `--scenario=fresh-restore-cold-cache-cold-store-isolated` is the new CLI surface. Updates land across the Rust orchestrator (`BenchmarkScenario`), `benchmarks/bench.sh`, the pacquet workflow, `benchmarks/README.md`, and `pacquet/CONTRIBUTING.md` so the names agree end-to-end. Adds a justified `#[allow(clippy::enum_variant_names)]` on the enum because every variant currently shares the `Fresh` prefix; the lint will stop firing once `Filled*`/`Resynced*` counterparts land. Bencher's stored history for the old benchmark names will become orphaned and can be archived in the UI.

github-actions · 2026-05-23T12:44:23Z

Micro-Benchmark Results

Linux

group                          main                                   pr
-----                          ----                                   --
tarball/download_dependency    1.00      8.4±0.22ms   517.2 KB/sec    1.00      8.4±0.20ms   514.7 KB/sec

codecov-commenter · 2026-05-23T12:44:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.80%. Comparing base (c5a1d08) to head (9471306).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11875      +/-   ##
==========================================
- Coverage   87.81%   87.80%   -0.02%     
==========================================
  Files         205      205              
  Lines       24429    24424       -5     
==========================================
- Hits        21453    21445       -8     
- Misses       2976     2979       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Reshapes the scenario identifiers so the linker mode is the leading group: `<linker>.<action>.<cache state>.<store state>`. Dots separate the four axes the bench varies, and `isolated-linker.*` / `gvs-linker.*` sort together in any dashboard that groups by prefix. Future buckets (`hoisted-linker.*`, `pnp-linker.*`) will slot in without disturbing the existing names. GVS is its own top-level bucket rather than a sub-variant of isolated — its perf profile differs enough to chart separately. Renames: - `clean-install` → `isolated-linker.fresh-install.cold-cache.cold-store` - `full-resolution` → `isolated-linker.fresh-install.hot-cache.hot-store` - `frozen-lockfile` → `isolated-linker.fresh-restore.cold-cache.cold-store` - `frozen-lockfile-hot-cache` → `isolated-linker.fresh-restore.hot-cache.hot-store` - `peek` → `isolated-linker.fresh-add-dep.hot-cache.hot-store` - `gvs-warm` → `gvs-linker.fresh-restore.hot-cache.hot-store` Each Rust variant now carries `#[value(name = "…")]` so clap accepts the dotted CLI form (`--scenario=isolated-linker.fresh-install.cold-cache.cold-store`). Display labels follow the slug structure: `Isolated linker: fresh install, cold cache + cold store` and `GVS linker: fresh restore, hot cache + hot store`. The `#[allow(clippy::enum_variant_names)]` is renewed; 5 of 6 variants share the `Isolated` prefix today. Once `Hoisted*` / `Pnp*` buckets land the lint will stop firing on its own.

The longer match-arm pattern produced by the linker-first rename exceeded the rustfmt width budget. Auto-format breaks the `&["install", "--frozen-lockfile"]` body onto its own line so the arm stays within the limit.

zkochan marked this pull request as ready for review May 23, 2026 12:10

coderabbitai Bot requested changes May 23, 2026

View reviewed changes

Comment thread .github/workflows/benchmark.yml Outdated

Comment thread benchmarks/bench.sh

coderabbitai Bot approved these changes May 23, 2026

View reviewed changes

zkochan added 2 commits May 23, 2026 14:53

style: apply rustfmt after scenario rename

9471306

The longer match-arm pattern produced by the linker-first rename exceeded the rustfmt width budget. Auto-format breaks the `&["install", "--frozen-lockfile"]` body onto its own line so the arm stays within the limit.

zkochan merged commit 4088de0 into main May 23, 2026
25 of 27 checks passed

zkochan deleted the bencher branch May 23, 2026 13:19

This was referenced May 23, 2026

ci(bencher): enforce PR thresholds and grant checks: write #11883

Merged

feat(registry): implement pnpm-registry server and adopt it in pacquet's test mock #11898

Merged

coderabbitai Bot mentioned this pull request Jun 20, 2026

refactor: move the TypeScript pnpm CLI into a pnpm11/ directory #12537

Merged

4 tasks

coderabbitai Bot mentioned this pull request Jun 29, 2026

fix(pnpr): stream proxied tarballs instead of buffering-then-verifying (and benchmark the cold pnpr cache) #12729

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

ci(bencher): record benchmark results to Bencher#11875

ci(bencher): record benchmark results to Bencher#11875
zkochan merged 6 commits into
mainfrom
bencher

zkochan commented May 23, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 23, 2026 •

edited

Loading

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

github-actions Bot commented May 23, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented May 23, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 23, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

zkochan commented May 23, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Required out-of-band setup

Out of scope

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

github-actions Bot commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integrated-Benchmark Report (Linux)

Scenario: Isolated linker: fresh restore, cold cache + cold store

Scenario: Isolated linker: fresh restore, hot cache + hot store

Scenario: Isolated linker: fresh install, cold cache + cold store

Scenario: Isolated linker: fresh install, hot cache + hot store

Uh oh!

qodo-code-review Bot commented May 23, 2026

Qodo reviews are paused for this user.

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Micro-Benchmark Results

Linux

Uh oh!

codecov-commenter commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zkochan commented May 23, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 23, 2026 •

edited

Loading

github-actions Bot commented May 23, 2026 •

edited

Loading

github-actions Bot commented May 23, 2026 •

edited

Loading

codecov-commenter commented May 23, 2026 •

edited

Loading