perf: Optimize npm lockfile parser#12093
Merged
Merged
Conversation
Cache all_dependencies results across workspace closures via a shared DashMap, reducing 1,758 redundant calls to ~350 unique lookups. Replace allocation-heavy possible_npm_deps with an inline buffer-reusing find_dep_in_lockfile. Switch NpmLockfile::packages from BTreeMap to HashMap for O(1) lookups with sorted serialization for deterministic output. Enumerate common NpmPackage fields explicitly to minimize serde(flatten) deserialization overhead.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Coverage Report
|
github-actions Bot
added a commit
that referenced
this pull request
Mar 2, 2026
## Release v2.8.13-canary.9 Versioned docs: https://v2-8-13-canary-9.turborepo.dev ### Changes - fix: Treat `npm: alias` dependencies as external, not workspace references (#12061) (`b179cb8`) - test: Port 18 more prysk tests to Rust (other/ + lockfile-aware-caching/) (#12062) (`7887af2`) - release(turborepo): 2.8.13-canary.8 (#12063) (`2a5522a`) - fix: Preserve file: protocol entries in pruned yarn v1 lockfile (#12064) (`ae5c1a1`) - perf: Use stack-allocated OidHash in FileHashes and skip expanded hashes on normal runs (#12065) (`677b248`) - test: Port all 8 find-turbo prysk tests to Rust (#12066) (`f827fca`) - fix: Support pnpm per-workspace lockfiles in turbo prune (#12067) (`23d047d`) - test: Port final 2 prysk tests to Rust (100% complete) (#12068) (`6d7e057`) - fix: Resolve Berry prune failure when resolutions contain patch overrides (#12069) (`6fe3c5e`) - test: Add lockfile fixture for yarn berry resolution pruning (issue #2791) (#12071) (`6cc1654`) - chore: Remove prysk test framework entirely (#12070) (`ed2d05a`) - refactor: Clean up test infrastructure and eliminate duplication (#12072) (`338911d`) - fix: Retain injected workspace package entries during pnpm lockfile pruning (#12073) (`acbe869`) - ci: Exclude turborepo-lsp and turborepo-schema-gen from test builds (#12075) (`4ce12e2`) - refactor: Clean up test infrastructure + improve test quality (#12074) (`4571f2b`) - ci: Remove redundant cargo build from coverage job (#12077) (`3c9bbe2`) - perf: Speed up lockfile test suite (#12078) (`20024df`) - ci: Remove integration test serialization (#12079) (`24d7c02`) - fix: Preserve `file:` and `link:` protocol entries in pruned bun lockfile (#12076) (`2635d9a`) - fix: Stop running unnecessary npm install in engines tests (#12081) (`24e4905`) - test: Add lockfile fixture for pnpm v9 injected workspace deps (issue #8243) (#12082) (`4d4929b`) - fix: Filter orphaned Yarn packageExtensions entries during lockfile pruning (#12084) (`68eb223`) - fix: Align experimentalObservability on object maps rather than arrays (#12089) (`9b9d1e4`) - examples: Upgrade with-react-native-web example to use latest versions (#12085) (`980ca43`) - fix: duplicate /signup? in Vercel URL (#12088) (`e865b51`) - ci: Deduplicate Rust test compilation with nextest archive (#12083) (`962cf39`) - fix: Prevent yarn integration tests from hanging on corepack prompts (#12090) (`29b0da7`) - fix: Prevent turbo dev from hanging when daemon file watching fails (#12091) (`b0d2f62`) - ci: Skip pnpm install for Rust test jobs (#12092) (`ebd137f`) - perf: Optimize npm lockfile parser (#12093) (`e4b4a66`) - chore: Trim unused dependency features for faster compilation (#12094) (`03b79e0`) - fix: Prevent lockfile-aware yarn test from hanging on corepack downloads (#12095) (`bf516e4`) - fix: Exclude turborepo-repository from JS smoke test in release workflow (#12097) (`fecc400`) --------- Co-authored-by: Turbobot <turbobot@vercel.com>
anthonyshew
added a commit
that referenced
this pull request
Mar 4, 2026
#12146) ## Summary - Fix `turbo prune` silently dropping deeply nested npm dependencies from the pruned lockfile - Add regression test that directly asserts the pruned lockfile retains all nested packages ## Problem When a workspace has dependencies installed deep under its `node_modules` (e.g. `packages/pkg1/node_modules/parent/node_modules/child`), `rehoist_packages` would double-process them. The parent entry's sub-dep relocation already moves the children to the hoisted path, but each child also had its own entry in `to_rehoist` that would then delete the just-relocated entry and try to promote from the original (now-gone) path. This was introduced in #12093 — the bug exists in the `rehoist_packages` logic itself rather than the `BTreeMap` → `HashMap` change, but the HashMap's non-deterministic iteration order made it consistently manifest. ## Fix Skip entries in `rehoist_packages` where the extracted `pkg_name` contains `/node_modules/`. These deeply nested packages are already handled by their parent's sub-dep relocation step. ## Testing - Rust unit test `test_subgraph_preserves_deeply_nested_workspace_deps` fails without the fix, passes with it - 164/164 `cargo test -p turborepo-lockfiles` - 28/28 `pnpm check-lockfiles --pm npm` Closes #12139
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
all_dependenciesresults across workspace closures via a sharedDashMap, reducing ~1,758 redundant calls to ~350 unique lookups on a 110-package monorepopossible_npm_deps(which allocates aVec<String>per dependency) withfind_dep_in_lockfilethat reuses a singleStringbufferNpmLockfile::packagesfromBTreeMaptoHashMapfor O(1) lookups, with a customSerializeimpl that sorts keys for deterministic outputNpmPackagefields explicitly (integrity,dev,link,engines,bin, etc.) to minimizeserde(flatten)deserialization overhead — the flatten fallback now only fires for truly unknown fieldsProfiling
On a 110-package npm monorepo (224KB lockfile),
all_dependenciesdropped from 7.6ms / 1,758 calls to below measurable threshold. Total tracing span count dropped 30.6% (3,432 → 2,383). These savings scale linearly with lockfile size.Testing
cargo test -p turborepo-lockfilescargo test -p turborepo-repositorypnpm check-lockfiles --pm npm