Skip to content

perf: Optimize npm lockfile parser#12093

Merged
anthonyshew merged 1 commit into
mainfrom
shew/optimize-lockfile-parser
Mar 2, 2026
Merged

perf: Optimize npm lockfile parser#12093
anthonyshew merged 1 commit into
mainfrom
shew/optimize-lockfile-parser

Conversation

@anthonyshew

Copy link
Copy Markdown
Contributor

Summary

  • Cache all_dependencies results across workspace closures via a shared DashMap, reducing ~1,758 redundant calls to ~350 unique lookups on a 110-package monorepo
  • Replace allocation-heavy possible_npm_deps (which allocates a Vec<String> per dependency) with find_dep_in_lockfile that reuses a single String buffer
  • Switch NpmLockfile::packages from BTreeMap to HashMap for O(1) lookups, with a custom Serialize impl that sorts keys for deterministic output
  • Enumerate common NpmPackage fields explicitly (integrity, dev, link, engines, bin, etc.) to minimize serde(flatten) deserialization overhead — the flatten fallback now only fires for truly unknown fields

Profiling

On a 110-package npm monorepo (224KB lockfile), all_dependencies dropped from 7.6ms / 1,758 calls to below measurable threshold. Total tracing span count dropped 30.6% (3,432 → 2,383). These savings scale linearly with lockfile size.

Testing

  • 163/163 cargo test -p turborepo-lockfiles
  • 128/128 cargo test -p turborepo-repository
  • 27/27 npm lockfile prune E2E tests via pnpm check-lockfiles --pm npm

Cache all_dependencies results across workspace closures via a shared
DashMap, reducing 1,758 redundant calls to ~350 unique lookups. Replace
allocation-heavy possible_npm_deps with an inline buffer-reusing
find_dep_in_lockfile. Switch NpmLockfile::packages from BTreeMap to
HashMap for O(1) lookups with sorted serialization for deterministic
output. Enumerate common NpmPackage fields explicitly to minimize
serde(flatten) deserialization overhead.
@anthonyshew anthonyshew requested a review from a team as a code owner March 2, 2026 05:30
@anthonyshew anthonyshew requested review from tknickman and removed request for a team March 2, 2026 05:30
@vercel

vercel Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
examples-basic-web Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-designsystem-docs Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-gatsby-web Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-kitchensink-blog Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-nonmonorepo Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-svelte-web Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-tailwind-web Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
examples-vite-web Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
turbo-site Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
turborepo-agents Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am
turborepo-test-coverage Building Building Preview, Comment, Open in v0 Mar 2, 2026 5:30am

@github-actions

github-actions Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

Metric Coverage
Lines 83.77%
Functions 79.69%
Branches 0.00%

View full report

@anthonyshew anthonyshew merged commit e4b4a66 into main Mar 2, 2026
55 checks passed
@anthonyshew anthonyshew deleted the shew/optimize-lockfile-parser branch March 2, 2026 05:42
github-actions Bot added a commit that referenced this pull request Mar 2, 2026
## Release v2.8.13-canary.9

Versioned docs: https://v2-8-13-canary-9.turborepo.dev

### Changes

- fix: Treat `npm: alias` dependencies as external, not workspace
references (#12061) (`b179cb8`)
- test: Port 18 more prysk tests to Rust (other/ +
lockfile-aware-caching/) (#12062) (`7887af2`)
- release(turborepo): 2.8.13-canary.8 (#12063) (`2a5522a`)
- fix: Preserve file: protocol entries in pruned yarn v1 lockfile
(#12064) (`ae5c1a1`)
- perf: Use stack-allocated OidHash in FileHashes and skip expanded
hashes on normal runs (#12065) (`677b248`)
- test: Port all 8 find-turbo prysk tests to Rust (#12066) (`f827fca`)
- fix: Support pnpm per-workspace lockfiles in turbo prune (#12067)
(`23d047d`)
- test: Port final 2 prysk tests to Rust (100% complete) (#12068)
(`6d7e057`)
- fix: Resolve Berry prune failure when resolutions contain patch
overrides (#12069) (`6fe3c5e`)
- test: Add lockfile fixture for yarn berry resolution pruning (issue
#2791) (#12071) (`6cc1654`)
- chore: Remove prysk test framework entirely (#12070) (`ed2d05a`)
- refactor: Clean up test infrastructure and eliminate duplication
(#12072) (`338911d`)
- fix: Retain injected workspace package entries during pnpm lockfile
pruning (#12073) (`acbe869`)
- ci: Exclude turborepo-lsp and turborepo-schema-gen from test builds
(#12075) (`4ce12e2`)
- refactor: Clean up test infrastructure + improve test quality (#12074)
(`4571f2b`)
- ci: Remove redundant cargo build from coverage job (#12077)
(`3c9bbe2`)
- perf: Speed up lockfile test suite (#12078) (`20024df`)
- ci: Remove integration test serialization (#12079) (`24d7c02`)
- fix: Preserve `file:` and `link:` protocol entries in pruned bun
lockfile (#12076) (`2635d9a`)
- fix: Stop running unnecessary npm install in engines tests (#12081)
(`24e4905`)
- test: Add lockfile fixture for pnpm v9 injected workspace deps (issue
#8243) (#12082) (`4d4929b`)
- fix: Filter orphaned Yarn packageExtensions entries during lockfile
pruning (#12084) (`68eb223`)
- fix: Align experimentalObservability on object maps rather than arrays
(#12089) (`9b9d1e4`)
- examples: Upgrade with-react-native-web example to use latest versions
(#12085) (`980ca43`)
- fix: duplicate /signup? in Vercel URL (#12088) (`e865b51`)
- ci: Deduplicate Rust test compilation with nextest archive (#12083)
(`962cf39`)
- fix: Prevent yarn integration tests from hanging on corepack prompts
(#12090) (`29b0da7`)
- fix: Prevent turbo dev from hanging when daemon file watching fails
(#12091) (`b0d2f62`)
- ci: Skip pnpm install for Rust test jobs (#12092) (`ebd137f`)
- perf: Optimize npm lockfile parser (#12093) (`e4b4a66`)
- chore: Trim unused dependency features for faster compilation (#12094)
(`03b79e0`)
- fix: Prevent lockfile-aware yarn test from hanging on corepack
downloads (#12095) (`bf516e4`)
- fix: Exclude turborepo-repository from JS smoke test in release
workflow (#12097) (`fecc400`)

---------

Co-authored-by: Turbobot <turbobot@vercel.com>
anthonyshew added a commit that referenced this pull request Mar 4, 2026
#12146)

## Summary

- Fix `turbo prune` silently dropping deeply nested npm dependencies
from the pruned lockfile
- Add regression test that directly asserts the pruned lockfile retains
all nested packages

## Problem

When a workspace has dependencies installed deep under its
`node_modules` (e.g.
`packages/pkg1/node_modules/parent/node_modules/child`),
`rehoist_packages` would double-process them. The parent entry's sub-dep
relocation already moves the children to the hoisted path, but each
child also had its own entry in `to_rehoist` that would then delete the
just-relocated entry and try to promote from the original (now-gone)
path.

This was introduced in #12093 — the bug exists in the `rehoist_packages`
logic itself rather than the `BTreeMap` → `HashMap` change, but the
HashMap's non-deterministic iteration order made it consistently
manifest.

## Fix

Skip entries in `rehoist_packages` where the extracted `pkg_name`
contains `/node_modules/`. These deeply nested packages are already
handled by their parent's sub-dep relocation step.

## Testing

- Rust unit test `test_subgraph_preserves_deeply_nested_workspace_deps`
fails without the fix, passes with it
- 164/164 `cargo test -p turborepo-lockfiles`
- 28/28 `pnpm check-lockfiles --pm npm`

Closes #12139
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant