You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On the fresh-resolve install path, the same package's tarball can be downloaded twice in a race window: once by the resolve-time prefetcher and again by CreateVirtualStore's cold batch. It is not a correctness bug — the CAFS is content-addressed and write_cas_file is idempotent — but it wastes bandwidth/CPU and (since #12236) can render the "Downloading X MB/Y MB" gauge twice for one package.
Why it happens
The prefetch → install coordination via tarball_mem_cache + Notify only ever applied to the old recursive InstallPackageFromRegistry path (which calls run_with_mem_cache). The phased CreateVirtualStore path bypasses it:
The resolver kicks off a background prefetch of package P — a full download + extract — detached via tokio::spawn, run_with_mem_cache::<Reporter>. The handle is dropped; nothing joins it.
Resolution finishes and CreateVirtualStore runs prefetch_cas_paths, which reads committed SQLite store-index rows. The store-index writer batches/commits asynchronously, so if P's row hasn't committed yet, P is classified cold.
The cold batch downloads P via InstallPackageBySnapshot → run_without_mem_cache, which consults the store index and the prefetched map but nottarball_mem_cache/the Notify. If P's row still isn't committed, it re-downloads — potentially while the prefetch is still streaming the same bytes.
CreateVirtualStore has no mem_cache field at all, so there is no path for the cold batch to observe an in-flight prefetch.
Impact
No corruption — idempotent content-addressed writes.
Wasted network + CPU for any package whose prefetch download hasn't committed its store-index row by the time the cold batch reaches it (most likely for packages resolved late and/or large tarballs that take a while to download).
Make the cold batch coordinate with in-flight prefetches instead of re-downloading. The natural fix is to thread tarball_mem_cache into CreateVirtualStore and have the cold-batch download go through run_with_mem_cache, so it picks up CacheValue::Available immediately or briefly blocks on the per-URL Notify — exactly the coordination the prefetching_resolver module doc already describes.
Constraint: no performance regression
The fix must not degrade install performance. In particular:
Do not add a blanket barrier that awaits all outstanding prefetch handles before materialization — that would serialize prefetch against the warm batch and kill the resolve/fetch pipelining the prefetcher exists to provide.
The cold batch should block only on the specific in-flight download for that one package (via the Notify), and only when it would otherwise re-download — warm packages and genuinely-cold (never-prefetched) packages must take exactly the path they do today.
The warm-batch rayon fast path must stay untouched.
A benchmark comparison (cold + warm fresh install, and --frozen-lockfile) against main should show no regression before merging.
Written by an agent (Claude Code, claude-opus-4-8).
Summary
On the fresh-resolve install path, the same package's tarball can be downloaded twice in a race window: once by the resolve-time prefetcher and again by
CreateVirtualStore's cold batch. It is not a correctness bug — the CAFS is content-addressed andwrite_cas_fileis idempotent — but it wastes bandwidth/CPU and (since #12236) can render the "Downloading X MB/Y MB" gauge twice for one package.Why it happens
The prefetch → install coordination via
tarball_mem_cache+Notifyonly ever applied to the old recursiveInstallPackageFromRegistrypath (which callsrun_with_mem_cache). The phasedCreateVirtualStorepath bypasses it:P— a full download + extract — detached viatokio::spawn,run_with_mem_cache::<Reporter>. The handle is dropped; nothing joins it.CreateVirtualStorerunsprefetch_cas_paths, which reads committed SQLite store-index rows. The store-index writer batches/commits asynchronously, so ifP's row hasn't committed yet,Pis classified cold.PviaInstallPackageBySnapshot→run_without_mem_cache, which consults the store index and the prefetched map but nottarball_mem_cache/theNotify. IfP's row still isn't committed, it re-downloads — potentially while the prefetch is still streaming the same bytes.CreateVirtualStorehas nomem_cachefield at all, so there is no path for the cold batch to observe an in-flight prefetch.Impact
fetchedstatus is still suppressed by the dedup set).Proposed fix
Make the cold batch coordinate with in-flight prefetches instead of re-downloading. The natural fix is to thread
tarball_mem_cacheintoCreateVirtualStoreand have the cold-batch download go throughrun_with_mem_cache, so it picks upCacheValue::Availableimmediately or briefly blocks on the per-URLNotify— exactly the coordination theprefetching_resolvermodule doc already describes.Constraint: no performance regression
The fix must not degrade install performance. In particular:
Notify), and only when it would otherwise re-download — warm packages and genuinely-cold (never-prefetched) packages must take exactly the path they do today.A benchmark comparison (cold + warm fresh install, and
--frozen-lockfile) againstmainshould show no regression before merging.Written by an agent (Claude Code, claude-opus-4-8).