Skip to content

perf(pnpr): stream resolved packages from /v1/resolve so the client overlaps tarball fetch with resolution #12234

Description

@zkochan

Re-scoped. The original framing of this issue ("pipeline downloads with resolution") was wrong: a frozen / pnpr install receives a complete lockfile, so there's no local tree-walk to interleave fetches with. The correct lever — below — is to make pnpr's resolve itself incremental, so the client can overlap fetch with server-side resolution. See the discussion in the comments.

Summary

POST /v1/resolve currently buffers the entire server-side resolution and returns the finished lockfile in one response. The client therefore can't start fetching any tarball until the whole graph is resolved.

But pnpr runs pacquet's resolver under the hood, which is incremental — it walks the dependency tree and produces packages one at a time (the same walk the native install wraps in PrefetchingResolver to fire downloads as each package resolves). So /v1/resolve can stream each resolved package as the walk yields it, and the client can fetch it immediately — giving the pnpr path the same fetch-overlaps-resolution shape the native path already has.

Why

path resolution overlap total
native (fresh) local tree-walk, incremental PrefetchingResolver fetches as each pkg resolves ≈ max(resolve, download)
pnpr (today) remote, buffered none — fetch waits for the full lockfile ≈ resolve + download
pnpr (streamed) remote, incremental client fetches as each pkg streams in ≈ max(resolve, download)

Streaming makes the pnpr path structurally identical to native (and potentially faster, since a warm box may resolve quicker than the client would locally).

Expected win (and the gating measurement)

The saving is exactly the server-side resolve time that currently blocks the first fetch:

saving ≈ min(server_resolve, downloads) ≈ server_resolve

Measured /v1/resolve durations on a 109-dep fixture:

  • warm box (packument cache hot): ~0.2 s → streaming saves ~0.2 s (negligible).
  • cold box (packument cache stale; default TTL is 5 min, so a benchmark that idles between runs hits this): ~3.5 s → streaming saves up to ~3.5 s.

Gate before building: log the actual /v1/resolve duration during a representative run. If it's a meaningful fraction of total install time (cold/distant box), streaming is the right fix. If it's ~0.2 s, the install-time gap is elsewhere (materialization / cold-batch path) and this change won't move it — investigate there instead.

Proposed design

  • /v1/resolve streams NDJSON frames as the resolver yields packages (fits the protocol's existing NDJSON error-frame shape): one frame per resolved package carrying what the client needs to fetch (name@version, integrity, tarball URL), plus a terminal done frame, and an E error frame if resolution aborts mid-stream.
  • Policy gates (minimumReleaseAge, trustPolicy) are already applied at pick-time during resolution, so each streamed package is pre-checked — no separate gate needed.
  • Input-lockfile verification still runs up front (before any package is streamed).
  • The client begins fetching each tarball (directly from the registry, per the resolve-only design in perf(pnpr): resolve server-side and fetch tarballs directly #12232) as its frame arrives, and assembles + writes pnpm-lock.yaml from the stream at the end.
  • Negotiated via the /-/pnpr handshake / a protocol-version bump so older clients fall back to the buffered response.

Acceptance

  • With a non-trivial server resolve time, the pnpr install overlaps tarball fetch with resolution and closes the wall-clock gap vs. a native install.
  • Older clients still work against the buffered path.

Companion: the reporting inconsistency that made this gap look like a redundant-download bug is #12235.


Written by an agent (Claude Code, claude-opus-4-8).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions