Skip to content

fix: installation should not fail when peer has no version#333

Merged
zkochan merged 1 commit into
masterfrom
issue293
Aug 31, 2016
Merged

fix: installation should not fail when peer has no version#333
zkochan merged 1 commit into
masterfrom
issue293

Conversation

@zkochan

@zkochan zkochan commented Aug 30, 2016

Copy link
Copy Markdown
Member

Expose the version when possible, but the installation shouldn't
fail even when one of the peer dependencies have no version.

close #293

zkochan added a commit that referenced this pull request Aug 30, 2016
Expose the version when possible, but the installation shouldn't
fail even when one of the peer dependencies have no version.

close #293, PR #333
Expose the version when possible, but the installation shouldn't
fail even when one of the peer dependencies have no version.

close #293, PR #333
@zkochan zkochan merged commit 1b82eec into master Aug 31, 2016
@zkochan zkochan deleted the issue293 branch September 3, 2016 20:28
pull Bot referenced this pull request in dwongdev/pnpm May 14, 2026
Resolves #330.

Mirrors pnpm v11's `@pnpm/bins.resolver`, `@pnpm/bins.linker`, and `@zkochan/cmd-shim` (resolved at pnpm `4750fd370c` and cmd-shim `0d79ca9534`):

- New `pacquet-cmd-shim` crate parses the `bin` field of a package manifest into commands. Both `bin: string | object` and the `directories.bin` glob fallback are supported, with the same path-traversal and URL-safe-name guards as upstream. Conflict resolution follows `pkgOwnsBin` + lexical compare.
- Shim generator writes all three flavors per bin: Unix `/bin/sh` (byte-compatible with pnpm's `generateShShim`), Windows `.cmd` (CRLF, `%~dp0` paths), and PowerShell `.ps1` (`$basedir`/`$exe` header with Windows/POSIX-pwsh detection). Mirrors `generateCmdShim` and `generatePwshShim` minus the unused `nodePath`/`prependToPath`/`progArgs` branches.
- All filesystem IO behind per-capability DI traits — `FsReadHead`, `FsReadFile`, `FsReadString`, `FsReadDir`, `FsWalkFiles`, `FsCreateDirAll`, `FsWrite`, `FsSetExecutable`, `FsEnsureExecutableBits` — following the pattern documented in [#332](pnpm/pacquet#332 (comment)). `FsWrite` is a thin shim over `std::fs::write` and is **not atomic**; a future atomic-write capability can build on it without renaming. Production callers turbofish the unit-struct provider `::<RealApi>`; tests inject unit-struct fakes to cover IO error paths real fs can't trigger portably. Generic param is named `Api` (not `Fs`) so the same provider can grow non-fs capabilities without a rename.
- New `LinkVirtualStoreBins` step walks each virtual-store slot and links its child packages' bins into the slot's own `node_modules/.bin`, matching `linkBinsOfDependencies` in pnpm's `building/during-install`. Slot iteration, per-slot child reads, and shim writes parallelised on rayon.
- `SymlinkDirectDependencies` and `InstallWithoutLockfile` call the bin linker after the symlink layout is in place. Direct-dep linking and the symlink loop are exposed as free `link_direct_dep_bins` / `symlink_direct_deps_into_node_modules` functions so they're unit-testable without the mock-registry scaffold.

Tests port the relevant `bins/resolver` and `bins/linker` cases from pnpm (`directories.bin` discovery, scoped bin names, dangerous locations, idempotent shim skip, conflict resolution) and add fakes-injected coverage for the IO error variants.

**Not in this PR**: hoisted-bin precedence and lifecycle-script-created bins. Both depend on subsystems pacquet hasn't built yet — hoisting (`hoistPattern` / `publicHoistPattern` resolution + the lift step) and lifecycle scripts (`exec/lifecycle/` runner + `allowBuilds` plumbing). The bin-linking side already does the right thing for both (shims are paths, so they pick up files generated post-extract; the precedence rule is a small addition once hoisting tags candidates as hoisted). Tracked separately in  with full porting coordinates.



## Performance

When the bin-linking step landed it added ~7% to the warm-cache Frozen Lockfile install ([benchmark](pnpm/pacquet#333 (comment))). Closing that gap took a sequence of changes that mirror how pnpm v11 avoids the same costs in its own `linkBinsOfDependencies` ([4750fd3](https://github.com/pnpm/pnpm/blob/4750fd370c/building/during-install/src/index.ts#L258-L309)):

- **Bundled manifests in `index.db`.** `extract_tarball_entries` captures the parsed `package.json`, picks the subset pnpm caches via a port of `normalizeBundledManifest` (`name`, `version`, `bin`, `directories`, `dependencies`, lifecycle `scripts`, …), and stuffs it into `PackageFilesIndex.manifest`. `prefetch_cas_paths` surfaces it back as a `HashMap<cache_key, Arc<Value>>` so the bin linker doesn't have to re-read `package.json` per child off disk.
- **msgpackr-records encoder learns `serde_json::Value`.** The encoder previously errored with `ManifestNotSupported` if anything ever set `manifest`; now it records-encodes nested JSON objects (slot-allocated per distinct key set) so `useRecords: true` readers see them as JS `Object`s rather than `Map`s — necessary for pnpm's `manifest.bin` / `manifest.directories?.bin` property accesses to resolve.
- **`Arc<Value>` for `PackageBinSource.manifest`.** The lockfile-driven bin-link path produces `slots × children` clones of the parsed manifest (~6k on the integrated-benchmark fixture). Sharing via `Arc` turns each push into a refcount bump.
- **`hasBin` filter via lockfile.** `LinkVirtualStoreBins` now takes the lockfile `packages:` section, pre-computes an `Option<HashSet<PackageKey>>` of `hasBin: true` keys at install start, and skips snapshot children that aren't in the set *before* any path-building or manifest lookup. ~95% of a real lockfile's packages don't declare a bin, so this short-circuits the bulk of the per-slot work. `None` is reserved for the pathological case where the lockfile lacks a `packages:` section (fall back to processing every child); `Some(empty)` is authoritative — lockfile says no package has a bin, every slot short-circuits. Mirrors pnpm's filter at `building/during-install/src/index.ts:283`.
- **Lockfile-driven slot iteration.** `run_lockfile_driven` walks `snapshots` directly and builds slot paths lexically. The previous `run_with_readdir` path enumerated every slot via `Api::read_dir(virtual_store_dir)` and probed each slot's own package directory with a wasted `read_dir` (~1267 wasted `open(O_DIRECTORY) + close` per warm install). The readdir-driven path survives as a fallback for `install_without_lockfile` and retains the existence probe there (small N, no virtual-store invariant from `create_virtual_dir_by_snapshot` to lean on).
- **Parallel prefetch decode.** `StoreIndex::get_many_raw` returns undecoded row bytes; the SQLite mutex releases before `prefetch_cas_paths` fans the msgpackr decode + integrity check across rayon.

Latest CI benchmark on this branch ([job 75462998251](https://github.com/pnpm/pacquet/actions/runs/25701679172/job/75462998251)):

| Scenario | pacquet@HEAD | pacquet@main | Relative |
|---|---|---|---|
| Frozen Lockfile | 2.074s ± 102ms | 2.026s ± 70ms | +2.4% (within noise) |
| Frozen Lockfile (Hot Cache) | **479.6ms ± 38ms** | **535.8ms ± 37ms** | **~10% faster than main** |

The follow-up pnpm#417 splits manifests into a dedicated `package_manifests` SQLite table so the unfiltered `package_index` prefetch payload stays at its original size and the manifest fetch becomes a targeted SELECT against just the `hasBin: true` subset.

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
pull Bot referenced this pull request in dwongdev/pnpm May 14, 2026
…op-level bin re-link (#342) (pnpm#523)

Two related bin-linking behaviors deferred from #333.

Behavior 1 — hoisted-bin precedence:
- Add `BinOrigin { Direct, Hoisted }` discriminator to `PackageBinSource`.
- New top tier in `pick_winner`: Direct wins outright over Hoisted
  regardless of ownership / lexical order. Mirrors upstream's
  `preferDirectCmds` partition at
  https://github.com/pnpm/pnpm/blob/4750fd370c/bins/linker/src/index.ts#L92.
- New `link_top_level_bins(modules_dir, direct, hoisted)` helper in
  `pacquet-package-manager` mixes both candidate lists into one
  `link_bins_of_packages` call so the new tier resolves conflicts in
  a single pass — previously the two passes (SymlinkDirectDependencies
  for direct + hoist pass for publicly-hoisted) wrote shims
  independently and a hoisted bin could shadow a direct one when its
  package name was lexically smaller.

Behavior 2 — lifecycle-script-created bins:
- Add a post-`BuildModules` per-importer top-level bin link pass.
  Re-reading each direct dep's `package.json` after lifecycle
  scripts run picks up bins generated by `postinstall` (the
  `@pnpm.e2e/generated-bins` upstream fixture). Idempotent for
  unchanged shims via `is_shim_pointing_at`.
- Mirrors upstream's `linkBinsOfImporter` pass that runs after
  `buildModules` at
  https://github.com/pnpm/pnpm/blob/4750fd370c/installing/deps-installer/src/install/index.ts#L1539.

Supporting changes:
- `PackageBinSource::new(location, manifest)` constructor + `with_origin`
  builder so existing call sites don't have to spell out the new field.
- Public `direct_dep_names_for_importer` helper extracted from
  `SymlinkDirectDependencies` so the post-build pass uses the same
  filter (skipped / first-wins / link_only) as the symlink phase.
- `InstallFrozenLockfileError::TopLevelBinLink` for the new failure
  surface.

Tests:
- `direct_origin_wins_over_hoisted_regardless_of_lexical` — pins the
  new tier overrides lexical ordering.
- `hoisted_origin_loses_to_existing_direct` — pins both arms of the
  new tier (Direct incumbent vs Hoisted candidate).
github-actions Bot pushed a commit to Eyalm321/pnpm that referenced this pull request May 18, 2026
Resolves pnpm#330.

Mirrors pnpm v11's `@pnpm/bins.resolver`, `@pnpm/bins.linker`, and `@zkochan/cmd-shim` (resolved at pnpm `4750fd370c` and cmd-shim `0d79ca9534`):

- New `pacquet-cmd-shim` crate parses the `bin` field of a package manifest into commands. Both `bin: string | object` and the `directories.bin` glob fallback are supported, with the same path-traversal and URL-safe-name guards as upstream. Conflict resolution follows `pkgOwnsBin` + lexical compare.
- Shim generator writes all three flavors per bin: Unix `/bin/sh` (byte-compatible with pnpm's `generateShShim`), Windows `.cmd` (CRLF, `%~dp0` paths), and PowerShell `.ps1` (`$basedir`/`$exe` header with Windows/POSIX-pwsh detection). Mirrors `generateCmdShim` and `generatePwshShim` minus the unused `nodePath`/`prependToPath`/`progArgs` branches.
- All filesystem IO behind per-capability DI traits — `FsReadHead`, `FsReadFile`, `FsReadString`, `FsReadDir`, `FsWalkFiles`, `FsCreateDirAll`, `FsWrite`, `FsSetExecutable`, `FsEnsureExecutableBits` — following the pattern documented in [pnpm#332](pnpm/pacquet#332 (comment)). `FsWrite` is a thin shim over `std::fs::write` and is **not atomic**; a future atomic-write capability can build on it without renaming. Production callers turbofish the unit-struct provider `::<RealApi>`; tests inject unit-struct fakes to cover IO error paths real fs can't trigger portably. Generic param is named `Api` (not `Fs`) so the same provider can grow non-fs capabilities without a rename.
- New `LinkVirtualStoreBins` step walks each virtual-store slot and links its child packages' bins into the slot's own `node_modules/.bin`, matching `linkBinsOfDependencies` in pnpm's `building/during-install`. Slot iteration, per-slot child reads, and shim writes parallelised on rayon.
- `SymlinkDirectDependencies` and `InstallWithoutLockfile` call the bin linker after the symlink layout is in place. Direct-dep linking and the symlink loop are exposed as free `link_direct_dep_bins` / `symlink_direct_deps_into_node_modules` functions so they're unit-testable without the mock-registry scaffold.

Tests port the relevant `bins/resolver` and `bins/linker` cases from pnpm (`directories.bin` discovery, scoped bin names, dangerous locations, idempotent shim skip, conflict resolution) and add fakes-injected coverage for the IO error variants.

**Not in this PR**: hoisted-bin precedence and lifecycle-script-created bins. Both depend on subsystems pacquet hasn't built yet — hoisting (`hoistPattern` / `publicHoistPattern` resolution + the lift step) and lifecycle scripts (`exec/lifecycle/` runner + `allowBuilds` plumbing). The bin-linking side already does the right thing for both (shims are paths, so they pick up files generated post-extract; the precedence rule is a small addition once hoisting tags candidates as hoisted). Tracked separately in  with full porting coordinates.



## Performance

When the bin-linking step landed it added ~7% to the warm-cache Frozen Lockfile install ([benchmark](pnpm/pacquet#333 (comment))). Closing that gap took a sequence of changes that mirror how pnpm v11 avoids the same costs in its own `linkBinsOfDependencies` ([4750fd3](https://github.com/pnpm/pnpm/blob/4750fd370c/building/during-install/src/index.ts#L258-L309)):

- **Bundled manifests in `index.db`.** `extract_tarball_entries` captures the parsed `package.json`, picks the subset pnpm caches via a port of `normalizeBundledManifest` (`name`, `version`, `bin`, `directories`, `dependencies`, lifecycle `scripts`, …), and stuffs it into `PackageFilesIndex.manifest`. `prefetch_cas_paths` surfaces it back as a `HashMap<cache_key, Arc<Value>>` so the bin linker doesn't have to re-read `package.json` per child off disk.
- **msgpackr-records encoder learns `serde_json::Value`.** The encoder previously errored with `ManifestNotSupported` if anything ever set `manifest`; now it records-encodes nested JSON objects (slot-allocated per distinct key set) so `useRecords: true` readers see them as JS `Object`s rather than `Map`s — necessary for pnpm's `manifest.bin` / `manifest.directories?.bin` property accesses to resolve.
- **`Arc<Value>` for `PackageBinSource.manifest`.** The lockfile-driven bin-link path produces `slots × children` clones of the parsed manifest (~6k on the integrated-benchmark fixture). Sharing via `Arc` turns each push into a refcount bump.
- **`hasBin` filter via lockfile.** `LinkVirtualStoreBins` now takes the lockfile `packages:` section, pre-computes an `Option<HashSet<PackageKey>>` of `hasBin: true` keys at install start, and skips snapshot children that aren't in the set *before* any path-building or manifest lookup. ~95% of a real lockfile's packages don't declare a bin, so this short-circuits the bulk of the per-slot work. `None` is reserved for the pathological case where the lockfile lacks a `packages:` section (fall back to processing every child); `Some(empty)` is authoritative — lockfile says no package has a bin, every slot short-circuits. Mirrors pnpm's filter at `building/during-install/src/index.ts:283`.
- **Lockfile-driven slot iteration.** `run_lockfile_driven` walks `snapshots` directly and builds slot paths lexically. The previous `run_with_readdir` path enumerated every slot via `Api::read_dir(virtual_store_dir)` and probed each slot's own package directory with a wasted `read_dir` (~1267 wasted `open(O_DIRECTORY) + close` per warm install). The readdir-driven path survives as a fallback for `install_without_lockfile` and retains the existence probe there (small N, no virtual-store invariant from `create_virtual_dir_by_snapshot` to lean on).
- **Parallel prefetch decode.** `StoreIndex::get_many_raw` returns undecoded row bytes; the SQLite mutex releases before `prefetch_cas_paths` fans the msgpackr decode + integrity check across rayon.

Latest CI benchmark on this branch ([job 75462998251](https://github.com/pnpm/pacquet/actions/runs/25701679172/job/75462998251)):

| Scenario | pacquet@HEAD | pacquet@main | Relative |
|---|---|---|---|
| Frozen Lockfile | 2.074s ± 102ms | 2.026s ± 70ms | +2.4% (within noise) |
| Frozen Lockfile (Hot Cache) | **479.6ms ± 38ms** | **535.8ms ± 37ms** | **~10% faster than main** |

The follow-up pnpm#417 splits manifests into a dedicated `package_manifests` SQLite table so the unfiltered `package_index` prefetch payload stays at its original size and the manifest fetch becomes a targeted SELECT against just the `hasBin: true` subset.

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
github-actions Bot pushed a commit to Eyalm321/pnpm that referenced this pull request May 18, 2026
…op-level bin re-link (pnpm#342) (pnpm#523)

Two related bin-linking behaviors deferred from pnpm#333.

Behavior 1 — hoisted-bin precedence:
- Add `BinOrigin { Direct, Hoisted }` discriminator to `PackageBinSource`.
- New top tier in `pick_winner`: Direct wins outright over Hoisted
  regardless of ownership / lexical order. Mirrors upstream's
  `preferDirectCmds` partition at
  https://github.com/pnpm/pnpm/blob/4750fd370c/bins/linker/src/index.ts#L92.
- New `link_top_level_bins(modules_dir, direct, hoisted)` helper in
  `pacquet-package-manager` mixes both candidate lists into one
  `link_bins_of_packages` call so the new tier resolves conflicts in
  a single pass — previously the two passes (SymlinkDirectDependencies
  for direct + hoist pass for publicly-hoisted) wrote shims
  independently and a hoisted bin could shadow a direct one when its
  package name was lexically smaller.

Behavior 2 — lifecycle-script-created bins:
- Add a post-`BuildModules` per-importer top-level bin link pass.
  Re-reading each direct dep's `package.json` after lifecycle
  scripts run picks up bins generated by `postinstall` (the
  `@pnpm.e2e/generated-bins` upstream fixture). Idempotent for
  unchanged shims via `is_shim_pointing_at`.
- Mirrors upstream's `linkBinsOfImporter` pass that runs after
  `buildModules` at
  https://github.com/pnpm/pnpm/blob/4750fd370c/installing/deps-installer/src/install/index.ts#L1539.

Supporting changes:
- `PackageBinSource::new(location, manifest)` constructor + `with_origin`
  builder so existing call sites don't have to spell out the new field.
- Public `direct_dep_names_for_importer` helper extracted from
  `SymlinkDirectDependencies` so the post-build pass uses the same
  filter (skipped / first-wins / link_only) as the symlink phase.
- `InstallFrozenLockfileError::TopLevelBinLink` for the new failure
  surface.

Tests:
- `direct_origin_wins_over_hoisted_regardless_of_lexical` — pins the
  new tier overrides lexical ordering.
- `hoisted_origin_loses_to_existing_direct` — pins both arms of the
  new tier (Direct incumbent vs Hoisted candidate).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resolved packages should have a version

1 participant