Skip to content

fix(pacquet): fresh install can download the same tarball twice (prefetch vs cold-batch race) #12241

Description

@zkochan

Summary

On the fresh-resolve install path, the same package's tarball can be downloaded twice in a race window: once by the resolve-time prefetcher and again by CreateVirtualStore's cold batch. It is not a correctness bug — the CAFS is content-addressed and write_cas_file is idempotent — but it wastes bandwidth/CPU and (since #12236) can render the "Downloading X MB/Y MB" gauge twice for one package.

Why it happens

The prefetch → install coordination via tarball_mem_cache + Notify only ever applied to the old recursive InstallPackageFromRegistry path (which calls run_with_mem_cache). The phased CreateVirtualStore path bypasses it:

  1. The resolver kicks off a background prefetch of package P — a full download + extract — detached via tokio::spawn, run_with_mem_cache::<Reporter>. The handle is dropped; nothing joins it.
  2. Resolution finishes and CreateVirtualStore runs prefetch_cas_paths, which reads committed SQLite store-index rows. The store-index writer batches/commits asynchronously, so if P's row hasn't committed yet, P is classified cold.
  3. The cold batch downloads P via InstallPackageBySnapshotrun_without_mem_cache, which consults the store index and the prefetched map but not tarball_mem_cache/the Notify. If P's row still isn't committed, it re-downloads — potentially while the prefetch is still streaming the same bytes.

CreateVirtualStore has no mem_cache field at all, so there is no path for the cold batch to observe an in-flight prefetch.

Impact

  • No corruption — idempotent content-addressed writes.
  • Wasted network + CPU for any package whose prefetch download hasn't committed its store-index row by the time the cold batch reaches it (most likely for packages resolved late and/or large tarballs that take a while to download).
  • Since fix(pacquet): report fresh-install prefetch downloads as fetched, not found_in_store #12236 routes the prefetch download through the real reporter, a racy re-download now shows the byte-progress gauge twice (the duplicate fetched status is still suppressed by the dedup set).

Proposed fix

Make the cold batch coordinate with in-flight prefetches instead of re-downloading. The natural fix is to thread tarball_mem_cache into CreateVirtualStore and have the cold-batch download go through run_with_mem_cache, so it picks up CacheValue::Available immediately or briefly blocks on the per-URL Notify — exactly the coordination the prefetching_resolver module doc already describes.

Constraint: no performance regression

The fix must not degrade install performance. In particular:

  • Do not add a blanket barrier that awaits all outstanding prefetch handles before materialization — that would serialize prefetch against the warm batch and kill the resolve/fetch pipelining the prefetcher exists to provide.
  • The cold batch should block only on the specific in-flight download for that one package (via the Notify), and only when it would otherwise re-download — warm packages and genuinely-cold (never-prefetched) packages must take exactly the path they do today.
  • The warm-batch rayon fast path must stay untouched.

A benchmark comparison (cold + warm fresh install, and --frozen-lockfile) against main should show no regression before merging.


Written by an agent (Claude Code, claude-opus-4-8).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions