ci(pnpr): benchmark the install accelerator under injected network latency#12166
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
🚧 Files skipped from review as they are similar to previous changes (6)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
📝 WalkthroughWalkthroughThis PR adds latency injection to the integrated benchmark harness: new CLI flags and workflow env vars, a TCP LatencyProxy that applies per-direction delays, and WorkEnv wiring to route pnpr and direct-registry traffic through optional proxies. ChangesLatency Injection for Integrated Benchmarks
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Review Summary by QodoInject network latency into pnpr install-accelerator benchmark
WalkthroughsDescription• Add latency-injecting TCP proxy to measure pnpr as remote service • Inject configurable round-trip latency between client and pnpr server • Inject configurable round-trip latency between client and registry • Enable fair pnpr-vs-direct performance comparison with network conditions Diagramflowchart LR
A["Benchmark CLI"] -->|"--pnpr-latency-ms"| B["LatencyProxy"]
B -->|"delayed connection"| C["pnpr Server"]
A -->|"--registry-latency-ms"| D["Registry LatencyProxy"]
D -->|"delayed connection"| E["Registry"]
C -->|"direct fast link"| E
A -->|"direct targets"| D
File Changes1. pacquet/tasks/integrated-benchmark/src/cli_args.rs
|
Contributor
Micro-Benchmark ResultsLinux |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the pacquet/tasks/integrated-benchmark harness to benchmark pnpr as a remote service by injecting configurable network latency between the benchmark client and (a) each pnpr@<rev> server and (b) the registry for direct-install targets, and wires those knobs into CI.
Changes:
- Add a dependency-free TCP latency proxy (
LatencyProxy) with a unit test validating RTT injection. - Add CLI flags
--pnpr-latency-msand--registry-latency-ms, and route direct targets via a registry-fronting proxy while keeping pnpr’s upstream registry “fast”. - Update the pacquet integrated benchmark GitHub Actions workflow to run with injected latency (50ms/50ms).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pacquet/tasks/integrated-benchmark/src/work_env.rs | Starts/owns latency proxies for pnpr servers and (optionally) the registry; threads registry selection through init. |
| pacquet/tasks/integrated-benchmark/src/main.rs | Plumbs new CLI args into WorkEnv. |
| pacquet/tasks/integrated-benchmark/src/cli_args.rs | Adds --pnpr-latency-ms and --registry-latency-ms flags. |
| pacquet/tasks/integrated-benchmark/src/latency_proxy.rs | Implements the latency-injecting TCP proxy. |
| pacquet/tasks/integrated-benchmark/src/latency_proxy/tests.rs | Adds a unit test asserting injected RTT is observed. |
| .github/workflows/pacquet-integrated-benchmark.yml | Enables latency injection in CI benchmark runs via env + flags. |
Comments suppressed due to low confidence (1)
pacquet/tasks/integrated-benchmark/src/work_env.rs:498
self.registry_latency_ms * 1000can overflowu64(panic in debug builds, wrap in release), producing an incorrect delay for large inputs. Usingsaturating_mul(orchecked_mulwith a clear error) avoids surprising behavior.
let one_way = Duration::from_micros(self.registry_latency_ms * 1000 / 2);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #12166 +/- ##
==========================================
- Coverage 87.58% 87.57% -0.01%
==========================================
Files 268 268
Lines 30775 30775
==========================================
- Hits 26953 26952 -1
- Misses 3822 3823 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
The pnpr install accelerator is a remote server, but the integrated benchmark ran it on loopback (RTT ~ 0), hiding the round-trip cost that dominates a real install — and that pnpr exists to reduce. Add a dependency-free, synchronous latency-injecting TCP proxy (`latency_proxy`) and two knobs: - `--pnpr-latency-ms`: fronts each `pnpr@<rev>` server, so the client<->server link pays the given round trip (half each direction). - `--registry-latency-ms`: fronts the registry for the direct (`pacquet`/`pnpm`/`--with-pnpm`) targets, so a direct install crosses the same network. `pnpr@<rev>` targets keep a direct (fast) registry link, modeling a warm, colocated server — so pnpr's advantage shows up as fewer round trips, not a faster backend. The workflow sets both equal (50ms) so the pnpr-vs-direct ratio is fair and the `pnpr` Bencher testbed becomes sensitive to protocol round-trip-count changes. Both default to 0 (current behavior); the registry proxy is also skipped in `--registry=npm` mode (already remote). Latency only — no bandwidth cap: the public registry is CDN-backed and CI runners are fast, so install time is latency/round-trip bound, not throughput bound. A bandwidth knob can follow if a slow-link scenario is ever wanted.
099f61c to
58fc5a6
Compare
Contributor
Integrated-Benchmark Report (Linux)Each scenario has pacquet rows (direct install) and pnpr rows (the same client through the pnpr install accelerator), so pnpr@HEAD vs pacquet@HEAD is the pnpr-vs-direct ratio. Cold-store scenarios wipe the client store between runs (warm server); hot-store scenarios keep it warm. The pacquet@HEAD rows feed the pacquet Bencher testbed; the pnpr@HEAD rows feed the pnpr testbed. Scenario: Isolated linker: fresh restore, cold cache + cold store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 4.804539941980001,
"stddev": 0.04574095780069623,
"median": 4.81695216608,
"user": 2.4701797,
"system": 3.69098138,
"min": 4.721375285080001,
"max": 4.879515463080001,
"times": [
4.879515463080001,
4.74345536408,
4.82238056208,
4.8073758810800005,
4.8126951170800005,
4.721375285080001,
4.82813746108,
4.83044944808,
4.8212092150800006,
4.77880562308
]
},
{
"command": "pacquet@main",
"mean": 4.8045925800800005,
"stddev": 0.024213079371425066,
"median": 4.805319431580001,
"user": 2.4819907,
"system": 3.674528679999999,
"min": 4.762582469080001,
"max": 4.85271756708,
"times": [
4.78459063108,
4.81413825008,
4.813488507080001,
4.762582469080001,
4.799473041080001,
4.821541664080001,
4.85271756708,
4.791524567080001,
4.79470328208,
4.81116582208
]
},
{
"command": "pnpr@HEAD",
"mean": 2.04782462178,
"stddev": 0.05677254118767356,
"median": 2.0423340655799995,
"user": 2.6591366,
"system": 3.2096054799999996,
"min": 1.9847900410800001,
"max": 2.1436514500799997,
"times": [
2.0958916150799998,
2.1436514500799997,
1.9934350650800001,
2.0037231360799996,
1.9847900410800001,
2.0432111200799996,
1.98784298408,
2.04145701108,
2.11508085808,
2.0691629370799998
]
},
{
"command": "pnpr@main",
"mean": 2.01382269028,
"stddev": 0.04677088584298047,
"median": 2.0061719500799997,
"user": 2.6720528999999997,
"system": 3.2267809799999996,
"min": 1.95024657708,
"max": 2.08711413808,
"times": [
2.04948848608,
2.07521457908,
1.97678454208,
2.08711413808,
2.03697723108,
1.99631367408,
2.01603022608,
1.9763945170800001,
1.97366293208,
1.95024657708
]
}
]
}Scenario: Isolated linker: fresh restore, hot cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.6917696473200001,
"stddev": 0.014159505683299649,
"median": 0.6856930678200001,
"user": 0.36240582,
"system": 1.36664934,
"min": 0.68121430232,
"max": 0.7264126443200001,
"times": [
0.7264126443200001,
0.68397530632,
0.6850571553200001,
0.68388408832,
0.69096742032,
0.68121430232,
0.68632898032,
0.6838150473200001,
0.68926840732,
0.70677312132
]
},
{
"command": "pacquet@main",
"mean": 0.6924316719200001,
"stddev": 0.01657358825643042,
"median": 0.69363538132,
"user": 0.37573942000000005,
"system": 1.3493860400000002,
"min": 0.6711280123200001,
"max": 0.71656442432,
"times": [
0.7147495823200001,
0.6711280123200001,
0.6814662233200001,
0.6900261293200001,
0.69742881432,
0.71656442432,
0.7047712193200001,
0.6797071303200001,
0.67123055032,
0.6972446333200001
]
},
{
"command": "pnpr@HEAD",
"mean": 0.7023659282200001,
"stddev": 0.0771082867453285,
"median": 0.6753603673200002,
"user": 0.36774762,
"system": 1.33230884,
"min": 0.65197884232,
"max": 0.90859760832,
"times": [
0.7245170913200001,
0.6644032123200001,
0.65197884232,
0.68887470232,
0.6552931323200001,
0.6730103393200001,
0.72367684832,
0.6777103953200001,
0.65559711032,
0.90859760832
]
},
{
"command": "pnpr@main",
"mean": 0.69539410642,
"stddev": 0.061279721402655966,
"median": 0.6752558443200001,
"user": 0.36736801999999996,
"system": 1.35317554,
"min": 0.66360296732,
"max": 0.86738109332,
"times": [
0.7005818693200001,
0.6753339113200001,
0.66360296732,
0.6688389533200001,
0.66790373832,
0.6751210723200001,
0.6751777773200001,
0.68337286632,
0.67662681532,
0.86738109332
]
}
]
}Scenario: Isolated linker: fresh install, cold cache + cold store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 2.17999385446,
"stddev": 0.05054916269501022,
"median": 2.17216350316,
"user": 3.49534246,
"system": 2.9905380999999998,
"min": 2.1166765461600003,
"max": 2.2817212991600004,
"times": [
2.17741559916,
2.19741826516,
2.2422039751600003,
2.16691140716,
2.1352241421600002,
2.2817212991600004,
2.15476901316,
2.1166765461600003,
2.1401058761600003,
2.1874924211600004
]
},
{
"command": "pacquet@main",
"mean": 2.18983354746,
"stddev": 0.024697767106819875,
"median": 2.18818150316,
"user": 3.50491266,
"system": 2.9974135,
"min": 2.1568590271600003,
"max": 2.23757621416,
"times": [
2.19043505716,
2.1806103361600004,
2.1605581591600003,
2.1924853251600003,
2.2227285831600003,
2.23757621416,
2.18941103116,
2.18071976616,
2.1568590271600003,
2.1869519751600004
]
},
{
"command": "pnpr@HEAD",
"mean": 2.1551693985600004,
"stddev": 0.024723937739989592,
"median": 2.15599890966,
"user": 3.46374256,
"system": 2.9749629,
"min": 2.10285216716,
"max": 2.1966437241600003,
"times": [
2.15379206416,
2.1612551781600002,
2.15229931116,
2.10285216716,
2.15820575516,
2.1750549301600004,
2.1369931221600003,
2.1685455151600004,
2.1460522181600004,
2.1966437241600003
]
},
{
"command": "pnpr@main",
"mean": 2.1455947341600003,
"stddev": 0.024801879635451704,
"median": 2.14085374816,
"user": 3.4721729600000004,
"system": 2.9821278,
"min": 2.11366755316,
"max": 2.1971680781600003,
"times": [
2.15613924116,
2.14167717716,
2.1971680781600003,
2.11366755316,
2.1317563781600004,
2.14779695216,
2.1388291781600004,
2.17141125316,
2.14003031916,
2.1174712111600003
]
}
]
}Scenario: Isolated linker: fresh install, hot cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 1.309692991,
"stddev": 0.01634270228831319,
"median": 1.3073612169,
"user": 1.3946382400000001,
"system": 1.7388861400000004,
"min": 1.2882152278999999,
"max": 1.3407407749,
"times": [
1.3407407749,
1.3112838309,
1.3136121128999998,
1.3317362368999999,
1.3034386028999998,
1.2965103428999998,
1.2964214589,
1.3136469649,
1.3013243569,
1.2882152278999999
]
},
{
"command": "pacquet@main",
"mean": 1.3291774965999996,
"stddev": 0.03374716973384547,
"median": 1.3216780339,
"user": 1.39615914,
"system": 1.76412604,
"min": 1.2914387298999999,
"max": 1.3897264929,
"times": [
1.3152736259,
1.3335319938999999,
1.3069194038999998,
1.3041030788999999,
1.3897264929,
1.2914387298999999,
1.3280824419,
1.3836121339,
1.3011987248999999,
1.3378883399
]
},
{
"command": "pnpr@HEAD",
"mean": 1.3130789584999998,
"stddev": 0.03717151524287036,
"median": 1.3092068204,
"user": 1.37254454,
"system": 1.7404448399999999,
"min": 1.2773541308999998,
"max": 1.4055500858999999,
"times": [
1.3068551129,
1.2815679298999998,
1.2829278548999998,
1.4055500858999999,
1.3261139289,
1.3115585279,
1.2773541308999998,
1.2972469639,
1.3287200238999999,
1.3128950258999998
]
},
{
"command": "pnpr@main",
"mean": 1.3345174006999998,
"stddev": 0.04167100872962534,
"median": 1.3185124129,
"user": 1.3708818400000002,
"system": 1.76575664,
"min": 1.2863828859,
"max": 1.4304151398999998,
"times": [
1.3055200108999998,
1.2863828859,
1.3161162239,
1.4304151398999998,
1.3716543089,
1.3541828348999998,
1.3352534199,
1.3135718569,
1.3111687238999998,
1.3209086019
]
}
]
} |
Contributor
|
Bencher Report| Branch | pr/12166 |
| Testbed | pacquet |
🚨 1 Alert
| Benchmark | Measure Units | View | Benchmark Result (Result Δ%) | Upper Boundary (Limit %) |
|---|---|---|---|---|
| isolated-linker.fresh-restore.cold-cache.cold-store | Latency seconds (s) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 4.80 s(+134.43%)Baseline: 2.05 s | 2.46 s (195.36%) |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| isolated-linker.fresh-install.cold-cache.cold-store | 📈 view plot 🚷 view threshold | 2,179.99 ms(-6.65%)Baseline: 2,335.33 ms | 2,802.40 ms (77.79%) |
| isolated-linker.fresh-install.hot-cache.hot-store | 📈 view plot 🚷 view threshold | 1,309.69 ms(-13.44%)Baseline: 1,512.98 ms | 1,815.57 ms (72.14%) |
| isolated-linker.fresh-restore.cold-cache.cold-store | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 4,804.54 ms(+134.43%)Baseline: 2,049.47 ms | 2,459.37 ms (195.36%) |
| isolated-linker.fresh-restore.hot-cache.hot-store | 📈 view plot 🚷 view threshold | 691.77 ms(+6.38%)Baseline: 650.26 ms | 780.32 ms (88.65%) |
This was referenced Jun 6, 2026
Merged
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The pnpr install accelerator is a remote server, but the integrated benchmark ran it on loopback (RTT ≈ 0), which hides the round-trip cost that dominates a real install — and that pnpr exists to reduce. This injects network latency so the benchmark measures pnpr as the remote service it is in production.
How
A dependency-free, synchronous latency-injecting TCP proxy (
latency_proxy) plus two knobs onintegrated-benchmark:--pnpr-latency-ms— fronts eachpnpr@<rev>server, so the client↔server link pays the given round trip (half each direction).--registry-latency-ms— fronts the registry for the direct (pacquet/pnpm/--with-pnpm) targets, so a direct install crosses the same network.pnpr@<rev>targets keep a direct (fast) registry link — that models a warm, colocated server, so pnpr's advantage shows up as fewer round trips, not a faster backend:The workflow sets both equal (
50ms) so the in-run pnpr-vs-direct ratio is fair and thepnprBencher testbed (pnpr@HEAD vs pnpr@main) becomes sensitive to protocol round-trip-count changes — which is what makes the upcoming protocol work (collapsing the 3-round-trip handshake/install/files flow) measurable on main. See #12165 for that plan.Notes
0(current behavior unchanged); the registry proxy is also skipped in--registry=npmmode (already remote).cargo check/clippy/fmt/dylintclean.Written by an agent (Claude Code, claude-opus-4-8).
Summary by CodeRabbit
New Features
Tests