Skip to content

fix(lockfile): preserve non-registry and bun platform entries#338

Merged
jdx merged 1 commit intomainfrom
fix/package-lock-git-resolved
Apr 27, 2026
Merged

fix(lockfile): preserve non-registry and bun platform entries#338
jdx merged 1 commit intomainfrom
fix/package-lock-git-resolved

Conversation

@jdx
Copy link
Copy Markdown
Contributor

@jdx jdx commented Apr 27, 2026

Summary

  • Parse git resolved URLs from package-lock.json entries as LocalSource::Git instead of registry packages.
  • Key those locked packages by the git source dep path so install imports the checkout rather than fetching <name>@<version> from npm.
  • Re-emit npm git resolved fields with the npm-compatible git+ prefix, while keeping unsupported non-git local sources out of package-lock output.
  • Accept scalar or array os / cpu / libc metadata in Bun lockfile package entries.
  • Add parser, writer, and BATS regressions covering these lockfile compatibility cases.

Root Cause

The npm lockfile reader only preserved resolved values that looked like HTTP tarball URLs. Git resolved URLs such as git+ssh://...#<sha> were dropped, leaving only <pkg>@<version>, which sent install down the npm registry tarball path. The npm writer also needed to preserve the external npm-compatible git+ form when writing those pinned git sources back.

The Bun reader assumed platform metadata was always serialized as arrays. Real Bun lockfiles can carry single platform values as scalars, for example { "os": "darwin", "cpu": "arm64" }, which caused parse failures or metadata loss.

Note: the related pnpm scalar os / cpu / libc fix is now on main via #337, so this PR keeps only the remaining package-lock and Bun fixes.

Validation

  • cargo fmt --check
  • cargo test -p aube-lockfile
  • cargo clippy --all-targets -- -D warnings
  • cargo build
  • mise run test:bats test/git_deps.bats

@jdx jdx force-pushed the fix/package-lock-git-resolved branch from f9948a0 to f1787b1 Compare April 27, 2026 12:11
@jdx jdx marked this pull request as ready for review April 27, 2026 12:13
@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Apr 27, 2026

Greptile Summary

This PR fixes two lockfile compatibility bugs: npm's package-lock.json parser now recognises git resolved URLs (e.g. git+ssh://...#<sha>) as LocalSource::Git instead of dropping them and falling back to the registry, and the Bun lockfile parser now tolerates scalar os/cpu/libc platform strings in addition to arrays. The writer side is updated to re-emit the pinned git resolved URL and insert git packages into the canonical map under their hash-based key so the hoist tree can locate them.

Confidence Score: 4/5

Safe to merge; no P0/P1 issues found — changes are targeted, well-tested, and the round-trip for git resolved URLs is verified by both unit and integration tests.

No critical or blocking defects identified. The canonical-map double-insertion for git packages (under both name@version from build_canonical_map and name@git+<hash> from the new loop) is intentional and consistent with how build_hoist_tree looks up packages. The Bun scalar fix is trivial. Score held at 4 rather than 5 because the git dep_path/canonical-map interaction is non-trivial and integration test coverage for git packages with transitive registry dependencies is absent.

crates/aube-lockfile/src/npm.rs — specifically the canonical map insertion loop and npm_resolved_field; worth a second read if git packages with transitive deps are a target use-case.

Important Files Changed

Filename Overview
crates/aube-lockfile/src/npm.rs Adds local_git_source_from_resolved to classify git resolved URLs as LocalSource::Git, keys their dep_paths accordingly, and emits the pinned URL back in npm_resolved_field; also inserts git packages into the canonical map under their hash-based key so the hoist/write path can find them.
crates/aube-lockfile/src/bun.rs Adds deserialize_with = "aube_util::string_or_seq" to os, cpu, and libc on RawBunMeta so scalar platform strings are accepted without breaking array deserialization; includes a focused unit test.
test/git_deps.bats Adds an integration test that builds a fake bare-git repo, writes a package-lock.json with a git+file:// resolved URL, runs aube install, and verifies the git checkout lands in node_modules.

Reviews (4): Last reviewed commit: "fix(lockfile): preserve non-registry and..." | Re-trigger Greptile

Comment thread crates/aube-lockfile/src/npm.rs
@jdx jdx force-pushed the fix/package-lock-git-resolved branch from f1787b1 to 78c5e34 Compare April 27, 2026 12:16
@jdx jdx changed the title fix(lockfile): preserve package-lock git resolved URLs fix(lockfile): preserve non-registry and platform entries Apr 27, 2026
@jdx jdx marked this pull request as draft April 27, 2026 12:17
@jdx jdx marked this pull request as ready for review April 27, 2026 12:17
@jdx jdx force-pushed the fix/package-lock-git-resolved branch from 78c5e34 to a48393a Compare April 27, 2026 12:24
@jdx jdx changed the title fix(lockfile): preserve non-registry and platform entries fix(lockfile): preserve non-registry and bun platform entries Apr 27, 2026
@jdx jdx marked this pull request as draft April 27, 2026 12:25
@jdx jdx marked this pull request as ready for review April 27, 2026 12:27
Comment thread crates/aube-lockfile/src/lib.rs Outdated
Comment thread crates/aube-lockfile/src/npm.rs
@jdx jdx force-pushed the fix/package-lock-git-resolved branch from a48393a to eecae7a Compare April 27, 2026 12:30
@jdx jdx marked this pull request as draft April 27, 2026 12:30
@jdx jdx marked this pull request as ready for review April 27, 2026 12:32
@jdx jdx force-pushed the fix/package-lock-git-resolved branch from eecae7a to 45f5a25 Compare April 27, 2026 12:35
@jdx jdx marked this pull request as draft April 27, 2026 12:36
@jdx jdx marked this pull request as ready for review April 27, 2026 12:38
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 45f5a25. Configure here.

Comment thread crates/aube-lockfile/src/npm.rs
jdx added a commit that referenced this pull request Apr 27, 2026
## Summary

- Preserve existing top-level `package.json` key order when commands
rewrite dependency sections.
- Parse Bun lockfile local tarball entries whose package ident omits the
`file:` prefix.
- Accept abbreviated git commit SHAs from existing Bun GitHub-shorthand
lockfile entries during checkout verification.

## Details

`aube add` already preserved manifest order by editing the parsed raw
`package.json` object and syncing only dependency sections. This PR
moves that targeted raw-JSON dependency-section writer into the shared
command helpers so `add`, `remove`, and `update --latest` use the same
order-preserving path.

`remove` still prunes pnpm/aube sidecar metadata, but now applies those
removals directly to the parsed raw JSON object instead of replacing
unrelated fields from the typed `PackageJson.extra` map. That keeps
existing key positions and avoids churn in unrelated or nested manifest
data.

The Bun regression matrix also exposed two plain-install failures not
covered by the currently open lockfile PRs:

- Bun can write local tarball package entries as
`local-helper@tarballs/local-helper-1.0.0.tgz` while the workspace
dependency remains `file:tarballs/local-helper-1.0.0.tgz`. Aube now
recognizes that prefixless `.tgz` ident as `LocalSource::Tarball`
instead of treating it as a registry version.
- Bun records GitHub shorthand lock entries with abbreviated commit IDs.
Aube's clone verifier now accepts a checked-out full SHA when it starts
with the abbreviated lockfile SHA, while still rejecting non-hex refs
and mismatched SHAs.

I checked the other open lockfile PRs before adding these: #337 covers
pnpm scalar platform fields, and #338 covers package-lock git resolved
URLs plus pnpm/Bun scalar platform metadata. Those fixes are
intentionally not duplicated here.

## Validation

- `cargo fmt --check`
- `cargo build`
- `cargo test -p aube write_manifest_dep_sections`
- `cargo test -p aube-lockfile test_parse_prefixless_local_tarball`
- `cargo test -p aube-lockfile test_parse_github_dep`
- `cargo test -p aube-store git_commit_matches_abbreviated_sha`
- `mise run test:bats test/remove.bats`
- `mise run test:bats test/update.bats`
- Bun regression matrix from
`johnpyp/2026-04-23-aube-bun-lock-regression-matrix` against
`target/debug/aube`; `github-shorthand` and `local-tarball` now pass
plain install as `plain-aube-unchanged`
- pre-commit hook: cargo fmt and cargo clippy for staged Rust files

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Medium risk because it changes how `package.json` is rewritten across
multiple commands and relaxes git checkout verification to accept
abbreviated SHAs, which could affect correctness of dependency updates
and git-sourced installs.
> 
> **Overview**
> Improves compatibility and reduces churn when working with Bun and
when editing `package.json`.
> 
> Commands that mutate dependencies (`add`, `remove`, `update --latest`)
now update only the dependency sections in the existing parsed
`package.json` object (via shared helpers) so **top-level key order is
preserved** and empty dep sections are removed without reserializing
unrelated fields; `remove` also prunes pnpm/aube sidecar metadata
directly in raw JSON to avoid reordering.
> 
> Bun lockfile parsing now treats prefixless local tarball idents (e.g.
`pkg@tarballs/foo.tgz` without `file:`) as `LocalSource::Tarball`, and
git dependency checkouts now accept **abbreviated hex SHAs** by
verifying the full HEAD starts with the requested short SHA. Added
targeted unit tests and a bats test to lock in these behaviors.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
edcf920. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
@jdx jdx force-pushed the fix/package-lock-git-resolved branch from 45f5a25 to a9c29ac Compare April 27, 2026 12:47
@jdx jdx merged commit 912463a into main Apr 27, 2026
18 checks passed
@jdx jdx deleted the fix/package-lock-git-resolved branch April 27, 2026 12:57
jdx added a commit that referenced this pull request Apr 30, 2026
## Summary

- For github / gitlab / bitbucket deps, re-derive an HTTPS URL from
`(host, owner, repo, sha)` at fetch time instead of dialing whatever
scheme the lockfile recorded. Matches npm `pacote` and pnpm
`gitHostedTarballFetcher`.
- SHA-pinned hosted reads go through
`https://codeload.github.com/<owner>/<repo>/tar.gz/<sha>` (no `git`
binary, no SSH key); on any HTTP error, fall back to a shallow `git
clone` over the rewritten HTTPS URL so a system git credential helper
(gh CLI etc.) keeps private repos working.
- Branch / tag committishes are pinned via `git ls-remote` on the same
rewritten HTTPS URL before reaching the codeload path. Non-hosted hosts
(self-hosted GitLab / Gitea / arbitrary) still use the URL as written,
preserving SSH-only setups.

## Why

Reported in [discussion
#335](#335) as a follow-up to
#338. After #338 made `package-lock.json` git deps parse correctly, aube
was dialing the lockfile-canonical `git+ssh://git@github.com/…#<sha>`
directly, which fails for users with HTTPS-only git auth (e.g. `gh`
CLI's git credential helper, no `~/.ssh/`). npm/pnpm never dial that SSH
URL — it's an identity form, not a fetch URL.

`LocalSource::Git.url` in the lockfile is preserved verbatim so
cross-tool round-trip with pnpm / npm / yarn is unaffected.

## Test plan

- [x] `cargo test -p aube-lockfile -p aube-store -p aube-resolver --lib`
(459 unit tests, all green)
- [x] `cargo clippy --all-targets -- -D warnings`
- [x] `cargo fmt --check`
- [x] `mise run test:bats test/git_deps.bats test/install.bats
test/package_lock_write.bats` (87 / 87 pass — clone fallback path
unchanged)
- [x] 5 new unit tests covering: `parse_hosted_git` across all clone-URL
forms (https / ssh / git+ / scp / `github:`), per-provider codeload URL
synthesis, `extract_codeload_tarball` wrapper-strip + caching, defense
against `..`/absolute symlink targets in tarball entries, short-commit
rejection.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes git dependency fetching/materialization and introduces new
tarball extraction logic, which can affect install reliability and
security if edge cases slip through. Mitigated by strict SHA gating,
cache reuse, and explicit unsafe-path/symlink defenses with clone
fallback.
> 
> **Overview**
> Git dependencies on GitHub/GitLab/Bitbucket are now resolved/installed
using npm/pnpm-like *hosted routing*: derive a canonical `(host, owner,
repo)` identity from the lockfile URL, pin refs via `git ls-remote` over
derived HTTPS, then prefer fetching a provider-specific codeload-style
tarball for full-SHA commits (with fallback to shallow `git clone`).
> 
> This adds a new codeload extraction cache and hardened tarball
extraction path in `aube-store` (wrapper-dir stripping, atomic cache
population, entry/path/symlink validation, and Windows symlink
handling), and wires the new flow through both the resolver
(`resolve_git_source` now takes an optional `RegistryClient`) and
installer while keeping the original lockfile `url` bytes unchanged for
cross-tool round-tripping.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
8bd9198. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant