ci(integrated-benchmark): scenarios without lockfiles#11838
Conversation
Wires `clean-install`, `full-resolution`, `peek`, and `gvs-warm` into `pacquet-integrated-benchmark.yml` so per-PR runs cover the same scenario set the manual `benchmark.yml` workflow already exercises via `benchmarks/bench.sh`. Requested for #11837, where the perf delta affects the resolution-bound scenarios (`firstInstall`, `withWarmCache`, `withWarmModules`, `updatedDependencies`) that the prior two-scenario set did not measure. Each scenario gets its own step with a 10 min hyperfine timeout (same rationale as the existing steps) and writes per-scenario report copies that the summary step concatenates into `SUMMARY.md`.
Keep only the two new no-lockfile scenarios (`clean-install`, `full-resolution`) on top of the existing `frozen-lockfile` and `frozen-lockfile-hot-cache`. #11837's perf change is in the fresh-lockfile install path, which only runs when resolution runs — i.e., exactly the no-lockfile scenarios. `peek` mutates an existing lockfile and `gvs-warm` is a frozen-lockfile variant; neither exercises the affected path, and including them only costs per-PR CI wall time.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
🧰 Additional context used📓 Path-based instructions (1)pacquet/**/*.rs📄 CodeRabbit inference engine (pacquet/AGENTS.md)
Files:
🧠 Learnings (1)📚 Learning: 2026-05-20T23:07:58.444ZApplied to files:
🔇 Additional comments (1)
📝 WalkthroughWalkthroughAdds CI steps for Clean Install and Full Resolution benchmarks and embeds their trimmed JSON reports into the generated summary; introduces an optional ChangesBenchmark Execution and Reporting
Workspace Manifest and Workspace Creation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Enables the pacquet integrated benchmark workflow to run and report additional scenarios that do not rely on lockfiles, expanding CI benchmark coverage beyond the existing frozen-lockfile scenarios.
Changes:
- Re-enabled the
clean-installintegrated-benchmark scenario in CI, with a step-level timeout and artifact copies. - Added a new CI step for the
full-resolutionintegrated-benchmark scenario, also with a timeout and artifact copies. - Extended the generated
SUMMARY.mdto include both new scenarios (markdown + JSON details).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The integrated-benchmark clones each pacquet revision's source tree into
`<bench_dir>/pacquet/`, which on the pnpm/pnpm monorepo includes upstream
test fixtures like
`workspace/project-manifest-reader/__fixtures__/invalid-package-json/package.json`
— intentionally malformed JSON used to exercise pnpm's manifest reader.
Without a `packages:` field, both pnpm's `findPackages.ts:28` and
pacquet's `crates/workspace/src/projects.rs:128` default to `[".", "**"]`,
so the fresh-resolve install path's `find_workspace_projects` walk
descends into the cloned source tree and trips on the bad fixture:
Error: pacquet_package_manifest::serialization_error
× installing dependencies
╰─▶ expected `,` or `}` at line 3 column 3
The walk only runs on the fresh-lockfile branch (`install.rs:628-630`),
which is why frozen-lockfile and frozen-lockfile-hot-cache stay green
while clean-install and full-resolution fail every time.
Pin `packages: ['.']` in the synthesized manifest so enumeration stays
at the workspace root. The benchmark's installs are single-project,
so this doesn't narrow anything the install actually needed to see.
Fixtures supplied via `--fixture-dir` that already declare `packages:`
keep their own value.
Micro-Benchmark ResultsLinux |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11838 +/- ##
==========================================
- Coverage 87.51% 87.50% -0.01%
==========================================
Files 204 204
Lines 24165 24371 +206
==========================================
+ Hits 21149 21327 +178
- Misses 3016 3044 +28 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Clean-install and full-resolution go through pacquet's fresh-resolve install path, which is currently ~3-5x slower than pnpm on the `alotta-files` fixture (#11832). Hyperfine's default 1 warmup + 10 timed runs across three benchmark targets (pacquet@HEAD, pacquet@main, system pnpm) projects to ~13 min wallclock for these two scenarios, putting the previous 10 min cap right on the edge. Doubling to 20 min keeps the per-step timeout meaningful as a stuck- install detector without losing CI time when the bench is healthy. The frozen-lockfile steps stay at 10 min — they don't traverse the slower fresh-resolve path.
Pacquet doesn't expose `--no-frozen-lockfile` (only `--frozen-lockfile`,
`--prefer-frozen-lockfile`, and `--no-prefer-frozen-lockfile`). Passing
it makes clap reject the install:
error: unexpected argument '--no-frozen-lockfile' found
tip: a similar argument exists: '--frozen-lockfile'
The flag was redundant for this scenario anyway: full-resolution starts
every iteration with no lockfile on disk (init() skips the lockfile when
`lockfile_enabled()` is false; cleanup removes it; `lockfile=false` in
the synthesized npmrc/workspace prevents writing one). With no lockfile
present the frozen path is unreachable regardless of the flag, so both
tools take fresh resolution by definition. Fold full-resolution into
clean-install's bare `install` arm.
Integrated-Benchmark Report (Linux)Scenario: Frozen Lockfile
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 3.432137791,
"stddev": 0.5131474158415051,
"median": 3.4728318377000003,
"user": 2.2141706599999997,
"system": 2.81793552,
"min": 2.7508148867,
"max": 4.3067551947,
"times": [
3.0623568537,
3.6228358757,
3.9016572137,
4.3067551947,
3.8663068227000004,
3.3327167787,
2.7508148867,
2.8408256107,
3.0241617767,
3.6129468967
]
},
{
"command": "pacquet@main",
"mean": 3.5374191417,
"stddev": 0.955396071191703,
"median": 3.1557366637,
"user": 2.19220246,
"system": 2.7995113199999997,
"min": 2.6386430677000003,
"max": 5.8513465057000005,
"times": [
3.9379318077,
3.2755520987,
4.2931606247000005,
3.4410383697,
5.8513465057000005,
3.0060125027,
2.6386430677000003,
2.8661582027000003,
3.0284270087,
3.0359212287
]
},
{
"command": "pnpm",
"mean": 4.6919697311,
"stddev": 0.3064921195362401,
"median": 4.5391302327,
"user": 6.19275546,
"system": 3.0948177199999996,
"min": 4.4595908297,
"max": 5.3993428467,
"times": [
4.8065853177,
4.4595908297,
4.4665542807000005,
4.5566583027,
4.8314335437,
5.3993428467,
4.939259502700001,
4.4647337997,
4.5216021627,
4.4739367247
]
}
]
}Scenario: Frozen Lockfile (Hot Cache)
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 1.0645509687,
"stddev": 0.2413425384534893,
"median": 0.9662441136,
"user": 0.31625896,
"system": 1.2475041799999997,
"min": 0.9347396261,
"max": 1.7280357271,
"times": [
1.0326715241,
0.9347396261,
0.9704129311,
1.0809227101,
0.9620752961,
0.9371096761,
0.9451971141,
0.9458568211,
1.7280357271,
1.1084882611
]
},
{
"command": "pacquet@main",
"mean": 1.3754049080000001,
"stddev": 0.6810200221010172,
"median": 0.9717427330999999,
"user": 0.31944016,
"system": 1.2542702799999998,
"min": 0.9189297301,
"max": 2.6995864591000003,
"times": [
2.6995864591000003,
2.4855410731000003,
1.6172381091,
1.2897171001,
0.9980939551,
0.9385917991,
0.9350819971,
0.9258773461,
0.9453915111,
0.9189297301
]
},
{
"command": "pnpm",
"mean": 3.084938983,
"stddev": 0.915259332031806,
"median": 2.7510851601,
"user": 2.3868880599999995,
"system": 1.64292318,
"min": 2.2192529021,
"max": 4.8155520831,
"times": [
4.8155520831,
2.7691458211,
2.6214956331,
2.2192529021,
2.7330244991000003,
2.4690061301,
2.4362879021,
4.6525392950999995,
3.2882912481,
2.8447943161
]
}
]
}Scenario: Clean Install
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 30.47734548176,
"stddev": 1.6576371408923283,
"median": 30.32630003036,
"user": 37.143677620000005,
"system": 6.436595079999999,
"min": 28.31951969686,
"max": 33.50848208986,
"times": [
28.31951969686,
30.45625080886,
32.34147678686,
30.19634925186,
28.99876924586,
33.50848208986,
31.39359800786,
29.085849294859997,
31.29807714386,
29.17508249086
]
},
{
"command": "pacquet@main",
"mean": 30.07770865406,
"stddev": 1.3096417988378966,
"median": 30.40315272736,
"user": 36.730923819999994,
"system": 6.43413398,
"min": 27.49071027586,
"max": 31.706271241859998,
"times": [
27.49071027586,
28.742119391859998,
29.07686431086,
30.798967282859998,
30.399892320859998,
29.80396184786,
30.40641313386,
31.38445084786,
30.96743588686,
31.706271241859998
]
},
{
"command": "pnpm",
"mean": 6.6810260132599995,
"stddev": 0.40647312340317987,
"median": 6.52462765986,
"user": 8.84956532,
"system": 3.5214553800000004,
"min": 6.22174590486,
"max": 7.58677980286,
"times": [
6.93556216286,
6.92150659086,
6.22174590486,
6.52445564486,
6.43459128186,
6.31546062386,
6.52479967486,
7.58677980286,
6.88434815986,
6.46101028586
]
}
]
}Scenario: Full Resolution
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 29.390766097939995,
"stddev": 2.381549581403614,
"median": 28.586513870939996,
"user": 35.94428694,
"system": 5.41284692,
"min": 26.92729011794,
"max": 34.35905947994,
"times": [
34.35905947994,
32.30388097894,
30.40443093994,
28.856626388939997,
26.92729011794,
27.897087176939998,
29.73119255894,
27.19584418194,
27.915847802939997,
28.316401352939998
]
},
{
"command": "pacquet@main",
"mean": 26.49727573494,
"stddev": 1.3379078094887558,
"median": 26.463745728939998,
"user": 32.15257064000001,
"system": 5.35436672,
"min": 24.22016352394,
"max": 28.49459493094,
"times": [
28.49459493094,
26.589255834939998,
27.567866696939998,
28.015991541939997,
26.73843733794,
26.338235622939997,
24.22016352394,
24.749807145939997,
25.94716726394,
26.31123744994
]
},
{
"command": "pnpm",
"mean": 4.17401299524,
"stddev": 0.7363504401433941,
"median": 3.92708770094,
"user": 4.269346239999999,
"system": 1.9943813199999993,
"min": 3.7136983849400003,
"max": 6.16373130594,
"times": [
3.8270794279400002,
4.01078211594,
3.8446492559400003,
4.34985362394,
3.7136983849400003,
3.74822444194,
6.16373130594,
4.33323201594,
4.00952614594,
3.73935323394
]
}
]
} |
Summary by CodeRabbit