chore: refresh benchmarks for v1.18.2#851
Conversation
📝 WalkthroughWalkthroughBenchmark results are refreshed with new data collected on 2026-06-09, aube toolchain updated to v1.18.2, and all warm/cold/install-test metrics recalculated. README documentation is updated to reflect the new benchmark ratios. ChangesBenchmark Results Refresh
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Greptile SummaryRefreshes benchmark data from aube 1.18.0 → 1.18.2 and updates the README
Confidence Score: 4/5Safe to merge as-is; the only open question is whether the pnpm/npm cold-cache spikes reflect real regressions or a noisy benchmark run. All changes are data-only (benchmark JSON + README copy). The README ratios are arithmetically correct against the new means. The warm-install and install-test numbers look reasonable and consistent. The cold-cache numbers for pnpm and npm are the outliers — they changed dramatically with high variance — which the PR description itself flags as worth investigating before landing. benchmarks/results.json — specifically the gvs-cold pnpm and npm entries with high stddev and large absolute regressions. Important Files Changed
Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile |
| "bun": { | ||
| "text": "5.777s ± 0.442s", | ||
| "mean": 5.776975449539999, | ||
| "stddev": 0.44177328485860695, | ||
| "min": 4.96066754994, | ||
| "max": 6.36779450894 | ||
| "text": "5.747s ± 0.202s", | ||
| "mean": 5.747125209120001, | ||
| "stddev": 0.20228616147832887, | ||
| "min": 5.458690011120001, | ||
| "max": 6.09460248712 | ||
| }, | ||
| "deno": { | ||
| "text": "8.153s ± 0.385s", | ||
| "mean": 8.152680589700001, | ||
| "stddev": 0.38522163142190247, | ||
| "min": 7.2359744523, | ||
| "max": 8.5316405233 | ||
| "text": "8.815s ± 0.324s", | ||
| "mean": 8.81505870046, | ||
| "stddev": 0.3240567157409274, | ||
| "min": 8.30310792076, | ||
| "max": 9.37489252876 | ||
| }, | ||
| "pnpm": { | ||
| "text": "15.870s ± 0.392s", | ||
| "mean": 15.87049447584, | ||
| "stddev": 0.39198277953927113, | ||
| "min": 15.39932277324, | ||
| "max": 16.208181022239998 | ||
| "text": "23.602s ± 6.550s", | ||
| "mean": 23.60153016548, | ||
| "stddev": 6.549877553224676, | ||
| "min": 18.20302976388, |
There was a problem hiding this comment.
The cold-cache numbers for npm (+227%, 9.5 s → 31.1 s) and pnpm (+49%, 15.9 s → 23.6 s) look unusually large, and pnpm's stddev of ±6.55 s on a 23.6 s mean gives a coefficient of variation of ~28% — far higher than any other cell in the file. This level of spread strongly suggests at least one outlier run (pnpm max was 33.6 s vs. min 18.2 s). Per the PR description's own guidance to "investigate before landing" if anything "looks wildly off," these two cells are the most obvious candidates for a re-run before the results get rolled into the next release-plz-pr.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
🤖 Refreshed benchmarks
benchmarks/results.jsonwas pinned to aube1.18.0; the workspace is now1.18.2. Re-ranmise run bench:bumpon the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regeneratedbenchmarks/results.jsonplus the READMEBENCH_RATIOSblock. The benchmark matrix pins aube's GVS mode vianpm_config_enable_global_virtual_store=true|false(the auto-synthesized env alias for theenableGlobalVirtualStoresetting), so GitHub Actions' inheritedCI=trueenvironment does not change whether aube runs with GVS enabled or disabled.Benchmark changes
Versions:
Public ratios: warm installs vs Bun 7x -> 7x; warm installs vs pnpm 9x -> 6x; repeat test vs Bun 5x -> 5x; repeat test vs pnpm 37x -> 29x.
Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.
Once merged to main, the updated bench results flow into the next
release-plz-prrun automatically.Generated by the
bench-refreshworkflow.Note
Low Risk
Documentation and benchmark artifact updates only; no runtime or install logic changes.
Overview
Re-runs the hermetic benchmark suite for aube 1.18.2 and commits the refreshed
benchmarks/results.json(timestamp, pinnedversions.aube, and timing stats for warm/cold install and repeatinstall && testscenarios).Updates the README
BENCH_RATIOSmarketing copy to match the new data: warm install vs pnpm 9x → 6x, repeat test vs pnpm 37x → 29x (vs Bun ratios unchanged at 7x and 5x). Absolute aube times are somewhat higher than the prior pin (e.g. warm 272ms → 383ms); cold-cache numbers for other managers also shifted, which the workflow flags for human review before merge.Reviewed by Cursor Bugbot for commit 20460a7. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit