bench: add baselines for each package manager x node_modules layout by Boshen · Pull Request #1176 · oxc-project/oxc-resolver

Boshen · 2026-05-28T08:25:21Z

Summary

Adds a new bench target that resolves the same 16-specifier workload against eight realistic monorepo fixtures, one per package manager × node_modules layout combination:

Combo	Linker	Layout
`npm-flat`	npm	flat hoisted
`pnpm-isolated`	pnpm	symlinked `.pnpm/` virtual store
`pnpm-hoisted`	pnpm `shamefullyHoist: true`	flat hoisted
`yarn-nm`	yarn berry `node-modules`	flat
`yarn-pnpm`	yarn berry `pnpm`	symlinked `.store/` virtual store
`yarn-pnp`	yarn berry `pnp`	no `node_modules/`, `.pnp.cjs` manifest
`bun-flat`	bun `hoisted`	flat
`bun-isolated`	bun `isolated`	symlinked `.bun/` virtual store

Each combo asserts correct resolution before timing — the resolved path must end with the package's intra-package file and pass through a layout-specific segment for the package name (/<pkg>/, /<pkg>@, /<pkg>-{npm,patch,virtual,workspace}-, plus the +/- scoped variants). This catches wrong resolutions across all 8 layouts using a single suffix per request.

Layout

Fixtures live under fixtures/bench-pm/<slug>/ as a nested monorepo so the resolver exercises ancestor node_modules walking from a depth-3 importer:

fixtures/bench-pm/<slug>/
  package.json + lockfile + PM config       # all committed
  apps/web/
    src/entry.js                            # shallow importer
    nested/deep/src/index.js                # deep importer
  packages/ui/  packages/utils/             # workspace deps

Lockfiles and PM configs are committed. node_modules/ and .pnp.cjs are gitignored and generated by the new just install-bench-fixtures recipe.

The new just benchmark-pm runs the bench (with --features yarn_pnp so all 8 combos are included). The existing just benchmark is scoped to --bench resolver so CodSpeed CI behavior is unchanged. CI is not touched in this PR — heavy fixtures and bun installation are deferred.

Local results (cold, single resolver, 16 resolves)

Combo	Time
`yarn-nm`	285 µs
`bun-flat`	280 µs
`npm-flat`	263 µs
`pnpm-hoisted`	426 µs
`pnpm-isolated`	424 µs
`bun-isolated`	415 µs
`yarn-pnpm`	518 µs
`yarn-pnp`	2.79 ms (manifest reload dominates cold; warm is comparable)

Flat layouts cluster ~270 µs; symlinked/isolated cluster ~420 µs (symlink canonicalize tax); PnP cold dominated by .pnp.cjs parse.

This is a baseline PR — follow-up PRs will introduce per-layout resolver optimizations measured against these numbers.

Adds a new bench target that resolves the same 16-specifier workload against eight realistic monorepo fixtures (npm-flat, pnpm-isolated, pnpm-hoisted, yarn-nm, yarn-pnpm, yarn-pnp, bun-flat, bun-isolated). Each combo asserts correct resolution before timing — the resolved path must end with the package's intra-package file and pass through a layout-specific segment for the package name. Fixtures live under fixtures/bench-pm/<slug>/ and ship only the package.json, lockfile, and PM config. node_modules / .pnp.cjs are generated by the new `just install-bench-fixtures` recipe. The existing `just benchmark` recipe is unchanged (scoped to the resolver bench to keep CodSpeed CI isolated); the new benches are run via `just benchmark-pm`.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4b8dc78fdc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

codspeed-hq · 2026-05-28T08:34:48Z

Merging this PR will degrade performance by 3.56%

❌ 1 regressed benchmark
✅ 12 untouched benchmarks
🆕 8 new benchmarks
⏩ 5 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
🆕	`pm/yarn-flat`	N/A	964 µs	N/A
🆕	`pm/yarn-isolated`	N/A	1.1 ms	N/A
🆕	`pm/npm-flat`	N/A	970.6 µs	N/A
🆕	`pm/bun-isolated`	N/A	1.1 ms	N/A
🆕	`pm/pnpm-isolated`	N/A	1.1 ms	N/A
🆕	`pm/pnpm-hoisted`	N/A	1.1 ms	N/A
🆕	`pm/bun-flat`	N/A	971 µs	N/A
🆕	`pm/yarn-pnp`	N/A	8.6 ms	N/A
❌	`resolver_memory[multi-thread]`	387.8 µs	402.1 µs	-3.56%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing bench-pm-fixtures (db2f83b) with main (bb0bf37)}

5 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

- Add `just` and `oven-sh/setup-bun@v2` so the benchmark job can run all four package managers needed by `just install-bench-fixtures`. - Build with `--features codspeed,yarn_pnp` so the yarn-pnp combo is included.

Previous lockfiles were generated before apps/web/package.json declared the workspace-internal deps (@bench/{ui,utils}, plus the external deps that yarn-pnp's strict resolution requires). Regenerated against the current package.json so `pnpm install` (which CI runs frozen) succeeds.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd0b119a50

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Aligns yarn combo slugs with the layout-based naming used by the other PMs (npm-flat, pnpm-isolated/hoisted, bun-flat/isolated). The nodeLinker is still node-modules and pnpm respectively; only the slug, package.json name, and Combo variants change.

…figs Replace 8 near-duplicate monorepos with: - fixtures/bench-pm/template/ — single shared monorepo (uses workspace:* deps) - fixtures/bench-pm/configs/<combo>/ — per-PM config snippets + the small set of files that genuinely differ (npm-flat needs "*" instead of "workspace:*"; yarn-* combos need packageManager: yarn@4.9.2) - fixtures/bench-pm/installs/<combo>/ — materialized monorepos (gitignored) just install-bench-fixtures now: rm -rf installs/, copies template/ then configs/<combo>/ into installs/<combo>/, installs each. Lockfiles regenerate each run (no committed lockfile, no --frozen-lockfile in CI). Bench code looks for fixtures under installs/<slug>/. Committed fixture file count: 88 -> 20.

Yarn berry enables "hardened mode" automatically on public PRs, which forbids lockfile regeneration. Other PMs default to frozen-lockfile in CI too. Commit each combo's lockfile alongside its PM config so `just install-bench-fixtures` can run as a non-mutating verify step.

- Justfile: bun install was guarded by `command -v bun && (...) || echo`, which swallowed any non-zero exit from `bun install` itself. Use if/then/else so a real install failure surfaces. - Justfile: drop `touch yarn.lock` — yarn.lock now ships in configs/yarn-*/ and is copied during materialization. - Bench docstrings: update fixture path (installs/<slug>/) and refresh slug references after the yarn-nm->yarn-flat / yarn-pnpm->yarn-isolated rename.

Drop the single-thread / cold / multi-thread split. Each combo now has one bench: build a fresh Resolver and fan the 16-request workload across worker threads via rayon — modelling a bundler's per-build resolver. The resolver construction is inside the timed body so PnP manifest parse, store discovery, and initial cache warmup all count. Result: 24 -> 8 named benches under `pm/<combo>`.

@Boshen

## 🤖 New release * `oxc_resolver`: 11.20.0 -> 11.21.0 * `oxc_resolver_napi`: 11.20.0 -> 11.21.0 <details><summary>Changelog</summary> ## `oxc_resolver` <blockquote> ## [11.21.0](v11.20.0...v11.21.0) - 2026-06-03 ### 🚀 Features - *(tsconfig)* support package.json imports field in extends ([#1199](#1199)) (by @Boshen) ### 🐛 Bug Fixes - *(tsconfig)* apply each referenced project's own `allowJs` ([#1198](#1198)) (by @shulaoda) - make symlink_metadata VPath-aware for Yarn PnP ([#1183](#1183)) (by @Boshen) ### ⚡ Performance - borrow relative main field instead of allocating a "./" prefix ([#1187](#1187)) (by @Boshen) - *(cache)* move package.json path into parse instead of cloning ([#1186](#1186)) (by @Boshen) - eliminate symlink stat syscalls by reusing canonicalization ([#1184](#1184)) (by @Boshen) - reduce resolution syscalls by unifying stat and lstat ([#1182](#1182)) (by @Boshen) ### 💼 Other - add baselines for each package manager x node_modules layout ([#1176](#1176)) (by @Boshen) ### Contributors * @shulaoda * @Boshen * @renovate[bot] </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). Co-authored-by: oxc-guard[bot] <276638029+oxc-guard[bot]@users.noreply.github.com>

chatgpt-codex-connector Bot reviewed May 28, 2026

View reviewed changes

Comment thread justfile Outdated

ci: include package-manager benches in CodSpeed run

387ac07

- Add `just` and `oven-sh/setup-bun@v2` so the benchmark job can run all four package managers needed by `just install-bench-fixtures`. - Build with `--features codspeed,yarn_pnp` so the yarn-pnp combo is included.

Boshen force-pushed the bench-pm-fixtures branch from 5631faa to 387ac07 Compare May 28, 2026 13:43

chatgpt-codex-connector Bot reviewed May 28, 2026

View reviewed changes

Comment thread fixtures/bench-pm/configs/pnpm-hoisted/pnpm-workspace.yaml

Boshen added 5 commits May 28, 2026 22:27

Boshen merged commit 8dad6c9 into main May 28, 2026
13 checks passed

Boshen deleted the bench-pm-fixtures branch May 28, 2026 15:24

oxc-guard Bot mentioned this pull request May 28, 2026

chore: release v11.21.0 #1178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bench: add baselines for each package manager x node_modules layout#1176

bench: add baselines for each package manager x node_modules layout#1176
Boshen merged 8 commits into
mainfrom
bench-pm-fixtures

Boshen commented May 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

codspeed-hq Bot commented May 28, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Boshen commented May 28, 2026

Summary

Layout

Local results (cold, single resolver, 16 resolves)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

codspeed-hq Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 3.56%

Performance Changes

Footnotes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codspeed-hq Bot commented May 28, 2026 •

edited

Loading