Conversation
Greptile SummaryThis PR refreshes benchmark data from aube 1.5.1 to 1.5.2, updating Confidence Score: 4/5Safe to merge after confirming the ci-warm ordering reversal is within expected benchmark noise. All benchmark values are internally consistent (values match rounded means, README ratios match JSON data). The single P2 observation — bun marginally ahead of aube in ci-warm — is within one standard deviation of measurement noise, but the PR description itself asks reviewers to investigate suspicious reversals before landing. benchmarks/results.json — specifically the ci-warm benchmark block where bun's point estimate (1205 ms) now slightly leads aube (1225 ms) Important Files Changed
Reviews (1): Last reviewed commit: "chore: refresh benchmarks" | Re-trigger Greptile |
| "key": "ci-warm", | ||
| "label": "CI install (warm cache, GVS disabled)", | ||
| "values": { | ||
| "aube": 2920, | ||
| "bun": 3396, | ||
| "pnpm": 4864, | ||
| "npm": 12296, | ||
| "yarn": 22250 | ||
| "aube": 1225, | ||
| "bun": 1205, | ||
| "pnpm": 2168, | ||
| "npm": 5699, | ||
| "yarn": 11501 | ||
| }, | ||
| "stats": { | ||
| "aube": { | ||
| "text": "2.920s ± 0.243s", | ||
| "mean": 2.91985430576, | ||
| "stddev": 0.24303922637128808, | ||
| "min": 2.43771276266, | ||
| "max": 3.21360930466 | ||
| "text": "1.225s ± 0.191s", | ||
| "mean": 1.22464908702, | ||
| "stddev": 0.19055271818273176, | ||
| "min": 0.80737233592, | ||
| "max": 1.4099390869200001 | ||
| }, | ||
| "bun": { | ||
| "text": "3.396s ± 0.315s", | ||
| "mean": 3.39636523454, | ||
| "stddev": 0.3146729135109249, | ||
| "min": 2.83716329634, | ||
| "max": 3.71710399034 | ||
| "text": "1.205s ± 0.148s", | ||
| "mean": 1.2049532691, | ||
| "stddev": 0.14808909607483306, | ||
| "min": 0.9771803725, | ||
| "max": 1.3821758875 | ||
| }, | ||
| "pnpm": { | ||
| "text": "4.864s ± 0.138s", | ||
| "mean": 4.8644633075199994, | ||
| "stddev": 0.13817009101181754, | ||
| "min": 4.67673544532, | ||
| "max": 5.03683336232 | ||
| "text": "2.168s ± 0.096s", | ||
| "mean": 2.1678183984399997, | ||
| "stddev": 0.09563425419932359, | ||
| "min": 2.01870140084, | ||
| "max": 2.24773223884 | ||
| }, | ||
| "npm": { | ||
| "text": "12.296s ± 0.624s", | ||
| "mean": 12.29602675817, | ||
| "stddev": 0.6239702633629706, | ||
| "min": 11.90817776592, | ||
| "max": 13.225758665919999 | ||
| "text": "5.699s ± 0.050s", | ||
| "mean": 5.698752955589999, | ||
| "stddev": 0.050147743212713876, | ||
| "min": 5.6559950703399995, | ||
| "max": 5.76224722034 | ||
| }, | ||
| "yarn": { | ||
| "text": "22.250s ± 0.361s", | ||
| "mean": 22.250436521570002, | ||
| "stddev": 0.3607184224580939, | ||
| "min": 21.94888791732, | ||
| "max": 22.695980812320002 | ||
| "text": "11.501s ± 0.180s", | ||
| "mean": 11.500933147780001, | ||
| "stddev": 0.17996507734138148, | ||
| "min": 11.29641011278, | ||
| "max": 11.735161938780001 | ||
| } | ||
| } | ||
| }, |
There was a problem hiding this comment.
ci-warm scenario
The CI install (warm cache, GVS disabled) point estimates have flipped: aube went from clearly leading (2920 ms vs bun's 3396 ms) to trailing by 20 ms (1225 ms vs 1205 ms). Given the large overlap in standard deviations (aube ±191 ms, bun ±148 ms), the two are statistically indistinguishable here, but the directional reversal may be worth a sanity check before landing — the PR description specifically flags "if anything looks wildly off vs. the previous release, investigate before landing."
Benchmark changesVersions:
Public ratios: warm installs vs Bun 4x -> 11x; warm installs vs pnpm 5x -> 16x.
914d970 vs 4326926 | aube/bun/pnpm | 3 scenarios | 3 runs | 500mbit/50ms | generated by Codex. |
🤖 Refreshed benchmarks
benchmarks/results.jsonwas pinned to aube1.5.1; the workspace is now1.5.2. Re-ranmise run bench:bumpon the hermetic Verdaccio registry (500mbit / 50ms per the mise task) and regeneratedbenchmarks/results.jsonplus the READMEBENCH_RATIOSblock. The benchmark matrix pins aube's GVS mode vianpm_config_enable_global_virtual_store=true|false(the auto-synthesized env alias for theenableGlobalVirtualStoresetting), so GitHub Actions' inheritedCI=trueenvironment does not change whether aube runs with GVS enabled or disabled.Benchmark changes
Versions:
Public ratios: warm installs vs Bun 4x -> 4x; warm installs vs pnpm 5x -> 7x; repeat test vs Bun 4x -> 4x; repeat test vs pnpm 14x -> 34x.
Review the numbers before merging — if anything looks wildly off vs. the previous release, investigate before landing. Hermetic proxy jitter or an npmjs uplink hiccup can occasionally skew results.
Once merged to main, the updated bench results flow into the next
release-plz-prrun automatically.Generated by the
bench-refreshworkflow.