Skip to content
This repository was archived by the owner on May 14, 2026. It is now read-only.
This repository was archived by the owner on May 14, 2026. It is now read-only.

Install git-hosted packages from pnpm-lock.yaml (frozen-lockfile) #436

@zkochan

Description

@zkochan

Background

pnpm install --frozen-lockfile against a lockfile that contains git-hosted dependencies fails today in pacquet. Roadmap (#299) Stage 1 lists "Installation of git-hosted packages" as a remaining checkbox; this issue scopes it to materializing git-hosted lockfile entries during a frozen-lockfile install. Resolution (parsing github:user/repo, git+ssh://..., semver-range git deps, etc.) lives in pnpm's resolving/git-resolver and is not needed for frozen install — the lockfile already pins the commit (or the tarball URL + integrity). Track resolution with Stage 2.

A pnpm-lock.yaml written for a git-hosted dependency can take one of two shapes:

  1. type: git resolution{ repo, commit }. pnpm shells out to git, clones, checks out the pinned commit, runs the package's prepare lifecycle if needed, then packlists the result into the CAS.
  2. Tarball resolution with gitHosted: true{ tarball, integrity?, gitHosted: true }. Used when the git host (GitHub / GitLab / Bitbucket) exposes a tarball endpoint. pnpm downloads the tarball through the normal HTTP path, but the contents still need preparePackage because the host's archive endpoint does not run lifecycle hooks. The CAS key is gitHostedStoreIndexKey(pkgId, { built }) rather than the integrity-based storeIndexKey, since the cached output depends on whether the build actually ran.

Both shapes are produced by resolving/git-resolver (index.ts:65-85): hosted + non-SSH → tarball with gitHosted: true; otherwise GitResolution. Pacquet has to handle both because either can land in pnpm-lock.yaml.

Today

Both call sites that dispatch on LockfileResolution reject Git outright:

  • crates/package-manager/src/install_package_by_snapshot.rs:110-115 returns UnsupportedResolution { resolution_kind: "git" }.
  • crates/package-manager/src/create_virtual_store.rs:513-520 returns the same error from the store-index lookup path.

Pacquet's crates/lockfile/src/resolution.rs already models GitResolution { repo, commit } correctly. The TarballResolution struct, however, is #[serde(deny_unknown_fields)] and lacks the gitHosted field. A real pnpm lockfile that pins a github-tarball resolution will therefore fail to deserialize with an "unknown field gitHosted" error, before pacquet even reaches the install dispatcher. That's a silent correctness gap independent of the install logic.

No git client, no preparePackage equivalent, no packlist, and no git-hosted store-index keying anywhere in the workspace.

Upstream references (pnpm v11 94240bc046)

  • Git fetcherfetching/git-fetcher/src/index.ts. git clone (or git init + git fetch --depth 1 origin <commit> when the host is in gitShallowHosts), git checkout <commit>, git rev-parse HEAD to verify, preparePackage, rimraf .git, packlist, addFilesFromDir.
  • Git-hosted tarball fetcherfetching/tarball-fetcher/src/gitHostedTarballFetcher.ts. Reuses remoteTarballFetcher for the download, then runs preparePackage on the extracted tree, and stores under a built or not-built CAS key. If the prepare step ends up packing the same file set as the raw tarball, the raw entry is promoted in place to skip the redundant rewrite.
  • preparePackageexec/prepare-package/src/index.ts:29-90. Decides whether the package needs building (scripts.prepare, or prepublish* + missing main file), then runs <pm>-install + prepare + prepublish* lifecycles. Errors with ERR_PNPM_PREPARE_PACKAGE on failure and GIT_DEP_PREPARE_NOT_ALLOWED if the package wants to build but isn't in allowBuilds.
  • Store index keyingstore/index/src/index.ts:60-95. pickStoreIndexKey returns gitHostedStoreIndexKey(pkgId, { built }) (= pkgId\tbuilt or pkgId\tnot-built) when gitHosted === true or when integrity is missing; integrity-keyed otherwise.
  • Lockfile-types annotationlockfile/types/src/index.ts:88-107. TarballResolution.gitHosted is optional; the loader back-fills it on entries whose URL matches a known git host, for backward compatibility with older lockfiles.

Plan

A. Lockfile types

  • Add git_hosted: Option<bool> to TarballResolution in crates/lockfile/src/resolution.rs (serde rename gitHosted, skip on None). The #[serde(deny_unknown_fields)] currently bites lockfiles produced by recent pnpm; this is a bug today regardless of the rest of this issue.
  • Backward-compat back-fill on read: when git_hosted is absent but the tarball URL parses to a known git host (codeload.github.com, gitlab.com, bitbucket.org, plus the gitlab.com/bitbucket.org archive endpoints), set it to true. Mirrors the loader's behavior described in lockfile/types/src/index.ts:97-107.
  • Store-index key selection: port pickStoreIndexKey semantics. For tarball resolutions with git_hosted: true or missing integrity, key by pkgId\t{built,not-built} instead of integrity\tpkgId. This lives near store_index_key in crates/store-dir (or wherever the current pacquet equivalent sits).

B. Git fetcher (handles LockfileResolution::Git)

  • New module — crates/git-fetcher or under crates/tarball if shared cafs plumbing makes sense there. Port fetching/git-fetcher/src/index.ts semantics:
    • Shell out to git. On Windows pass -c core.longpaths=true.
    • When the host is in gitShallowHosts: git initgit remote add origin <repo>git fetch --depth 1 origin <commit>. Else: git clone <repo> <tempDir>.
    • git checkout <commit> and git rev-parse HEAD to verify the resolved commit matches the pinned one. Surface GIT_CHECKOUT_FAILED on mismatch.
    • Run preparePackage (see section D). Surface ERR_PNPM_PREPARE_PACKAGE on failure.
    • Delete .git before computing CAS contents.
    • packlist-equivalent: read .npmignore + files field + npm's default ignore rules to determine what gets stored.
    • Import the resulting file tree into the CAS via the existing pacquet addFilesFromDir-equivalent in crates/tarball / crates/store-dir (port or reuse).
  • Replace the UnsupportedResolution return at install_package_by_snapshot.rs:110-115 with a dispatch to the new fetcher.
  • Replace the same at create_virtual_store.rs:513-520 so the store-index key path also handles git resolutions (returns the gitHostedStoreIndexKey).
  • Wire a new git_shallow_hosts: Vec<String> to Config and pnpm-workspace.yaml parsing. Default per upstream (currently ['github.com', 'gist.github.com', 'gitlab.com', 'bitbucket.org', 'bitbucket.com'] — verify against config/reader defaults).
  • Surface a clear error if git is not on PATH. Don't try to bundle git.

C. Git-hosted tarball fetcher (handles TarballResolution { git_hosted: true })

  • When the install dispatcher encounters TarballResolution { git_hosted: true }, take a different post-download path:
    1. Download the tarball via the existing pacquet tarball fetcher into a raw CAS slot keyed under a \traw suffix on the files-index path (matching upstream's ${filesIndexFile}\traw at gitHostedTarballFetcher.ts:27).
    2. Extract into a temp dir, run preparePackage on it.
    3. packlist the prepared result.
    4. If the prepared file set is identical to the raw set, promote the raw entry to the final key in place (the optimization at gitHostedTarballFetcher.ts:88-100 — fast path for prepare-less git-hosted tarballs).
    5. Otherwise re-import the prepared tree under the regular files-index path.
  • Key under gitHostedStoreIndexKey(pkgId, { built }), not integrity. The built flag reflects whether preparePackage actually ran a build (it returns shouldBeBuilt).
  • Propagate the raw tarball's integrity into the lockfile result so future installs still pin the bytes from the host.

D. preparePackage

  • Port exec/prepare-package/src/index.ts. Order of operations: read package.json of the extracted tree, decide via packageShouldBeBuilt (any non-empty scripts.prepare, or any prepublish* script + missing main), then run <pm>-install + prepare + each non-empty PREPUBLISH_SCRIPTS (exec/prepare-package/src/index.ts:48-80) via pacquet's existing lifecycle-script runner.
  • Honor allowBuild — pacquet's AllowBuildPolicy already exists (commit 6af026fa); pass it in. Throw GIT_DEP_PREPARE_NOT_ALLOWED when the package wants to build but isn't allowlisted, mirroring upstream's error code + hint.
  • Detect the preferred package manager for the temp dir (lockfile sniffing — pnpm-lock.yaml / yarn.lock / package-lock.json); default to npm if none. Matters because the <pm>-install script is what materializes the prepare-time dependencies.
  • rimraf node_modules after the prepare finishes — those installed deps must not end up in the CAS.

E. Tests

  • Frozen-lockfile install of a fixture lockfile that pins a type: git resolution against a small public repo. Assert the snapshot lands in CAS and resolves via a virtual-store symlink. Use a fixture under crates/testing-utils/src/fixtures/ and a local repo (no network), per existing conventions.
  • Frozen-lockfile install of a TarballResolution { git_hosted: true } entry; assert the tarball goes through the prepare path and the CAS uses the gitHostedStoreIndexKey shape.
  • Lockfile round-trip: a pnpm-written lockfile containing both shapes loads, serializes, and re-loads without losing gitHosted. Closes the deny_unknown_fields gap.
  • Commit-mismatch case: a GitResolution whose commit no longer exists in the remote → GIT_CHECKOUT_FAILED.
  • Prepare-disallowed case: a git-hosted package with scripts.prepare not in allowBuildsGIT_DEP_PREPARE_NOT_ALLOWED.
  • Prepare-allowed case: same package with allowBuilds: { <name>: true } → install succeeds, post-build artifacts land in CAS, requires_build flag is set.
  • Shallow-clone gating: when the host is in git_shallow_hosts, assert that git fetch --depth 1 was used rather than full clone (the integration-style test can spy on git invocations via PATH shimming, mirroring upstream's test approach).
  • Port from plans/TEST_PORTING.md:563-572 once the implementation is in place. Mark the items there as done.

Out of scope (Stage 2 / follow-ups)

  • Resolution. resolving/git-resolver (parsing github:user/repo and friends, resolving committish to SHA, choosing tarball vs git resolution). Needed for non-frozen pnpm install and for pnpm add github:user/repo. Track with Stage 2.
  • isRepoPublic / SSH discovery. network/git-utils is mostly used by pnpm publish, not install.
  • hosted-git-info port. Required by the resolver, not the fetcher.
  • GVS (Add global virtual store support to pacquet install --frozen-lockfile #432). Git-hosted packages live in CAS the same way; the only interaction is the store-index key still being gitHostedStoreIndexKey. No extra work here.
  • Partial install (Partial install with --frozen-lockfile: read+write node_modules/.pnpm/lock.yaml and skip unchanged snapshots #433). Same per-snapshot skip logic applies — current_packages[depPath] plus isIntegrityEqual on the resolution. Git resolutions compare by { repo, commit }; git-hosted tarball resolutions compare by tarball URL + built flag.

Related


Written by an agent (Claude Code, claude-opus-4-7).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions