Skip to content

perf(install): pipeline per-project materialize into fetch phase#527

Merged
jdx merged 5 commits intoendevco:mainfrom
imjustprism:perf-materializer-fix
May 6, 2026
Merged

perf(install): pipeline per-project materialize into fetch phase#527
jdx merged 5 commits intoendevco:mainfrom
imjustprism:perf-materializer-fix

Conversation

@imjustprism
Copy link
Copy Markdown
Contributor

@imjustprism imjustprism commented May 6, 2026 •
edited by jdx
Loading

Changed

  • pipeline per-project (GVS-off) materialize work into fetch phase. each fetched (canonical_key, PackageIndex) triggers materialize_into against .aube/<dep_path>/ so by the time fetch finishes the dedicated link phase only creates top-level node_modules/<name> symlinks
  • aube-linker: new pub ensure_in_aube_dir wrapping materialize_into with idempotent entry-exists short-circuit
  • aube-linker: mkdirp exposed pub for use by the install-side materializer driver

Fixed

  • swap Vec<JoinHandle> for JoinSet in materializer driver. on early-return the set drops all in-flight tasks instead of detaching them, no orphan disk writes racing install cleanup
  • semaphore acquire unwrap replaced with proper miette error map. no panic across cross-crate boundary if the semaphore is closed
  • removed redundant mkdirp(parent) in ensure_in_aube_dir. materialize_into already batches create_dir_all for every parent it needs, the entry parent is just aube_dir which the driver pre-creates

Benchmarks

Ran the PR benchmark matrix locally against both the base commit df72dcbe9554e2e277ad85ba5135079d28bcbc54 and PR head 451b082d2c4818e41487abccc542e9c4e9348896 with the same host, same benchmark settings, and same tool versions:

RESULTS_JSON=<output.json> \
BENCH_TOOLS=aube,bun,pnpm \
BENCH_SCENARIOS=gvs-warm,ci-warm,ci-cold \
BENCH_PHASES=0 \
RUNS=3 \
WARMUP=1 \
BENCH_HERMETIC=1 \
BENCH_BANDWIDTH=500mbit \
BENCH_LATENCY=50ms \
mise x npm:pnpm@latest bun@latest node@24 -- bash benchmarks/bench.sh

cargo build --release completed before each benchmark run. pnpm was installed through mise's npm backend because mise x aqua:pnpm/pnpm@latest failed on Linux before the benchmark started with a missing pnpm-linux-x64 asset lookup.

Versions:

  • aube: 1.9.0
  • bun: 1.3.13
  • pnpm: 10.33.4
  • node: 24.15.0

PR head results

Scenario aube bun pnpm vs pnpm vs bun
Fresh install (warm cache) 0.171s ± 0.002s 0.422s ± 0.005s 1.041s ± 0.030s 6.1x faster 2.5x faster
CI install (warm cache, GVS disabled) 0.251s ± 0.005s 0.410s ± 0.003s 1.025s ± 0.025s 4.1x faster 1.6x faster
CI install (cold cache, GVS disabled) 1.490s ± 0.133s 1.408s ± 0.022s 2.085s ± 0.040s 1.4x faster 1.1x slower

Same-host base comparison

Public ratios: warm installs vs Bun 2x -> 2x; warm installs vs pnpm 5x -> 6x.

Benchmark aube bun pnpm
Fresh install (warm cache) 189ms -> 171ms (-10%) 419ms -> 422ms (+1%) 998ms -> 1041ms (+4%)
CI install (warm cache, GVS disabled) 271ms -> 251ms (-7%) 412ms -> 410ms (0%) 1019ms -> 1025ms (+1%)
CI install (cold cache, GVS disabled) 1538ms -> 1490ms (-3%) 1428ms -> 1408ms (-1%) 2060ms -> 2085ms (+1%)

Additional GVS cold check

gvs-cold is the cold-cache scenario with GVS enabled. It uses BENCH_SCENARIOS=gvs-cold and pins aube with npm_config_enable_global_virtual_store=true.

Benchmark aube bun pnpm
Fresh install (cold cache, GVS enabled) 1775ms -> 1303ms (-27%) 1419ms -> 1431ms (+1%) 2036ms -> 2091ms (+3%)

On PR head, aube is 1.6x faster than pnpm and 1.1x faster than Bun in this scenario.

Benchmark section generated by Codex.

@imjustprism imjustprism marked this pull request as draft May 6, 2026 15:01
@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 6, 2026 •
edited
Loading

Greptile Summary

This PR pipelines per-project (non-GVS) materialization into the fetch phase: each freshly-fetched package is materialized into .aube/<dep_path>/ while the next download is in flight, so the dedicated link phase only needs to create top-level node_modules/<name> symlinks.

  • Adds run_aube_dir_materializer with a JoinSet-backed concurrency model and inline error back-pressure (try_join_next inside the recv loop), replacing the old drain-and-discard approach for non-GVS installs.
  • Exposes ensure_in_aube_dir and mkdirp as pub in aube-linker; skips Cached packages from the materializer channel since the link phase already fast-paths them via pkg_nm_dir.exists().
  • Flips tarball streaming from opt-in (AUBE_TARBALL_STREAM=1) to opt-out (AUBE_DISABLE_TARBALL_STREAM=1) in both install call-sites.

Confidence Score: 5/5

Safe to merge; no currently-reachable incorrect behavior was found in the changed code paths.

The new materializer correctly uses JoinSet for abort-on-drop semantics and the semaphore acquire is properly propagated as an error. The main concern — whether canonical_to_contextualized is actually consulted — looks like dead allocation rather than a missing materialization, because each dep_path arrives individually on the channel. The TOCTOU in ensure_in_aube_dir is safe under the single-dispatch-per-dep_path invariant the caller upholds.

The canonical_to_contextualized logic in mod.rs deserves a second look if future work changes how the fetch loop batches packages by canonical key.

Important Files Changed

Filename Overview
crates/aube-linker/src/lib.rs Adds pub ensure_in_aube_dir (idempotent per-project materialization) and exposes mkdirp as pub; logic is straightforward, TOCTOU in the exists-check is benign with hardlink semantics
crates/aube/src/commands/install/lifecycle.rs Comment-only change: updates the tarball-streaming doc from opt-in to default-on; no logic changes in this file
crates/aube/src/commands/install/mod.rs Adds run_aube_dir_materializer for per-project pipelining; builds canonical_to_contextualized that appears unreachable given the channel always emits actual dep_paths; streaming and per-project materializer logic are otherwise sound

Reviews (2): Last reviewed commit: "[autofix.ci] apply automated fixes" | Re-trigger Greptile

@imjustprism imjustprism marked this pull request as ready for review May 6, 2026 15:37
@jdx jdx merged commit 9a02ecc into endevco:main May 6, 2026
20 checks passed
@greptile-apps greptile-apps Bot mentioned this pull request May 6, 2026
Merged
@chenrui333 chenrui333 mentioned this pull request May 7, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@imjustprism @jdx