perf: migrate internal cache and index files to MessagePack serialization by zkochan · Pull Request #10500 · pnpm/pnpm

zkochan · 2026-01-21T22:45:59Z

install with hot cache/hot store (pnpm i --ignore-scripts --offline --frozen-lockfile)

this is with JSON:

Time (mean ± σ):      9.975 s ±  1.587 s    [User: 2.372 s, System: 10.641 s]
Range (min … max):    6.926 s … 11.772 s    10 runs

this is with msgpackr:

Time (mean ± σ):      9.192 s ±  1.389 s    [User: 2.484 s, System: 10.590 s]
Range (min … max):    7.056 s … 10.959 s    10 runs

this is with msgpackr+structured data+maps in index files:

Time (mean ± σ):      8.048 s ±  0.910 s    [User: 2.458 s, System: 10.598 s]
Range (min … max):    7.257 s …  9.700 s    10 runs

install with cold cache (pnpm i --ignore-scripts)

this is with JSON:

  Time (mean ± σ):     28.375 s ±  4.509 s    [User: 9.215 s, System: 25.523 s]
  Range (min … max):   20.383 s … 34.349 s    10 runs

this is with msgpackr+structured:

  Time (mean ± σ):     24.176 s ±  3.430 s    [User: 10.060 s, System: 25.615 s]
  Range (min … max):   20.400 s … 31.497 s    10 runs

zkochan · 2026-01-22T00:25:09Z

cc @gluxon

gluxon · 2026-01-22T00:41:27Z

Interesting idea! I'll build this PR and run some performance benchmarks on our large monorepo.

zkochan · 2026-01-22T01:03:40Z

Pushed some more optimizations.

Not all tests are fixed yet but I think it works now and can be benchmarked.

…tion

gluxon · 2026-01-22T20:33:17Z

The results here ended up being surprising and I think there's something we missed in #10409.

Using a hot cache and hot store, perf seems to be about the same before and after this PR on a large repo. This is the same repo I tested in #10409 (comment). Installs using MessagePack and JSON end up being around ~60s.

However, I would have expected this to be in the ~30s range. This is how fast the same command takes on pnpm 10.28.0 and the commit right before switching to v8 buffers takes (d392c3d).

I assumed commit da112f7 would make installs go back to the ~30 range, but that's not the case. It's still around ~60s. I'm going to look at CPU profile more deeply.

Benchmarking MessagePack

git restore pnpm-lock.yaml
pnpm install --ignore-scripts

MessagePack (`e841247`)

Benchmark 1: node /Volumes/git/pnpm/pnpm/lib/pnpm.js install --ignore-scripts
  Time (mean ± σ):     61.974 s ±  1.106 s    [User: 89.470 s, System: 24.945 s]
  Range (min … max):   60.400 s … 63.764 s    10 runs

Benchmark 1: node /Volumes/git/pnpm/pnpm/lib/pnpm.js install --ignore-scripts
  Time (mean ± σ):     60.410 s ±  0.636 s    [User: 88.219 s, System: 24.682 s]
  Range (min … max):   59.449 s … 61.446 s    10 runs

Benchmark 1: node /Volumes/git/pnpm/pnpm/lib/pnpm.js install --ignore-scripts
  Time (mean ± σ):     61.564 s ±  0.991 s    [User: 88.228 s, System: 24.688 s]
  Range (min … max):   60.447 s … 63.410 s    10 runs

main (`c494de3`)

Benchmark 1: node /Volumes/git/pnpm/pnpm/lib/pnpm.js install --ignore-scripts
  Time (mean ± σ):     60.747 s ±  0.934 s    [User: 77.740 s, System: 25.461 s]
  Range (min … max):   58.928 s … 61.847 s    10 runs

Benchmark 1: node /Volumes/git/pnpm/pnpm/lib/pnpm.js install --ignore-scripts
  Time (mean ± σ):     61.225 s ±  2.055 s    [User: 76.852 s, System: 25.906 s]
  Range (min … max):   57.646 s … 64.484 s    10 runs

Benchmark 1: node /Volumes/git/pnpm/pnpm/lib/pnpm.js install --ignore-scripts
  Time (mean ± σ):     61.463 s ±  1.624 s    [User: 78.088 s, System: 25.850 s]
  Range (min … max):   58.600 s … 63.567 s    10 runs

zkochan · 2026-01-22T20:42:16Z

That's weird. I was running my benchmarks on this package.json: https://github.com/pnpm/pnpm.io/blob/main/benchmarks/fixtures/alotta-files/package.json

but I was only comparing before and after this change now. Wasn't comparing to v10.

I use this script for testing hot cache:

hyperfine --warmup 1 --runs 10 \
                                         --setup 'rm -rf node_modules _cache_ _store_ && node ~/src/pnpm/pnpm3/pnpm/dist/pnpm.mjs i --ignore-scripts' \
                                         --prepare 'rm -rf node_modules' \
                                         --export-json pnpm3-1-3.json \
                                         'node ~/src/pnpm/pnpm3/pnpm/dist/pnpm.mjs i --ignore-scripts --offline'

This one to test with cold cache:

hyperfine --warmup 1 --runs 10 \
                                         --prepare 'rm -rf node_modules _cache_ _store_ pnpm-lock.yaml' \
                                         --export-json pnpm2-cold-1.json \
                                         'node ~/src/pnpm/pnpm2/pnpm/dist/pnpm.mjs i --ignore-scripts'

This is what I got with hot cache using pnpm v10:

  Time (mean ± σ):     11.968 s ±  1.305 s    [User: 2.940 s, System: 10.719 s]
  Range (min … max):   10.270 s … 13.942 s    10 runs

so according to my benchmarks this change improves performance by a lot.

resuls with v8-file

Benchmark 1: node ~/src/pnpm/pnpm2/pnpm/dist/pnpm.mjs i --ignore-scripts --offline
  Time (mean ± σ):     15.769 s ±  1.070 s    [User: 2.770 s, System: 12.294 s]
  Range (min … max):   13.652 s … 17.076 s    10 runs

zkochan · 2026-01-22T22:44:41Z

@gluxon how does this look like with your fix?

gluxon · 2026-01-22T22:54:35Z

I'll check now! For others following along, the performance regression I was seeing with pnpm v11 alpha and v10 should be fixed by #10502.

Copilot

Pull request overview

Migrates pnpm’s internal store index files and metadata cache files from JSON serialization to MessagePack (via msgpackr) to improve install performance.

Changes:

Introduce @pnpm/fs.msgpack-file utilities for MessagePack read/write and adopt them across the repo.
Switch store index files and metadata cache files to .mpk, and update related readers/writers and CLI commands.
Replace object-based PackageFiles*/SideEffects* shapes with Map-based representations and update tests accordingly.

Reviewed changes

Copilot reviewed 72 out of 73 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
worker/tsconfig.json	Adds reference to new msgpack fs package.
worker/src/start.ts	Reads/writes package index via msgpack and migrates file maps to `Map`.
worker/package.json	Adds `@pnpm/fs.msgpack-file`, removes `load-json-file`.
store/plugin-commands-store/tsconfig.json	Adds reference to msgpack fs package.
store/plugin-commands-store/src/storeStatus/index.ts	Reads integrity index as `.mpk` and adapts `files` to `Map`.
store/plugin-commands-store/package.json	Adds msgpack fs dep, removes `load-json-file`.
store/plugin-commands-store-inspecting/tsconfig.json	Adds reference to msgpack fs package.
store/plugin-commands-store-inspecting/test/findHash.ts	Updates expected index extension to `.mpk`.
store/plugin-commands-store-inspecting/src/findHash.ts	Reads `.mpk` indexes and iterates `Map` entries.
store/plugin-commands-store-inspecting/src/catIndex.ts	Reads `.mpk` package index files.
store/plugin-commands-store-inspecting/package.json	Adds msgpack fs dep, removes `load-json-file`.
store/package-store/tsconfig.json	Adds reference to msgpack fs package.
store/package-store/src/storeController/prune.ts	Prunes `.mpk` index files and uses `Map` accessors.
store/package-store/package.json	Adds msgpack fs dep, removes `load-json-file`.
store/cafs/test/index.ts	Updates CAFS tests to use `Map` for files.
store/cafs/src/readManifestFromStore.ts	Uses `Map.get()` for `package.json` entry.
store/cafs/src/index.ts	Updates exported CAFS types to new `Map`-based types.
store/cafs/src/getFilePathInCafs.ts	Changes CAFS index path suffixes to `.mpk`.
store/cafs/src/checkPkgFilesIntegrity.ts	Updates integrity checks to iterate `Map`-based structures.
store/cafs-types/src/index.ts	Removes raw record types; standardizes on `Map` types.
reviewing/plugin-commands-licenses/test/snapshots/index.ts.snap	Updates snapshot header link.
reviewing/license-scanner/tsconfig.json	Adds reference to msgpack fs package.
reviewing/license-scanner/test/getPkgInfo.spec.ts	Updates expected missing-index path to `.mpk`.
reviewing/license-scanner/src/getPkgInfo.ts	Reads `.mpk` indexes and adapts license scanning to `Map`.
reviewing/license-scanner/package.json	Adds msgpack fs dep, removes `load-json-file`.
resolving/npm-resolver/tsconfig.json	Adds reference to msgpack fs package.
resolving/npm-resolver/test/utils/index.ts	Updates retry helper to read `.mpk` metadata cache.
resolving/npm-resolver/test/resolveJsr.test.ts	Updates tests to read cached meta from `.mpk`.
resolving/npm-resolver/test/index.ts	Updates tests to read cached meta from `.mpk`.
resolving/npm-resolver/src/pickPackage.ts	Reads/writes cached registry metadata as `.mpk`.
resolving/npm-resolver/package.json	Adds msgpack fs dep; adjusts `load-json-file` usage for tests.
pnpm/tsconfig.json	Adds reference to msgpack fs package.
pnpm/test/utils/retryLoadJsonFile.ts	Removes JSON retry helper.
pnpm/test/utils/index.ts	Stops exporting removed JSON retry helper.
pnpm/test/install/misc.ts	Updates tests to read/write integrity indexes as `.mpk` and use `Map`.
pnpm/package.json	Adds `@pnpm/fs.msgpack-file` dependency.
pnpm-workspace.yaml	Adds `msgpackr` to catalog and allows `msgpackr-extract` build.
pnpm-lock.yaml	Locks `msgpackr`(+extract) and wires new workspace package.
pkg-manager/package-requester/tsconfig.json	Adds reference to msgpack fs package.
pkg-manager/package-requester/test/index.ts	Updates tests to read `.mpk` indexes and use `Map`.
pkg-manager/package-requester/src/packageRequester.ts	Renames package-dir index files to `integrity*.mpk`.
pkg-manager/package-requester/package.json	Adds msgpack fs dependency.
pkg-manager/headless/tsconfig.json	Adds reference to msgpack fs package.
pkg-manager/headless/test/index.ts	Updates side-effects cache tests for `.mpk` + `Map` structures.
pkg-manager/headless/package.json	Adds msgpack fs dependency.
pkg-manager/core/tsconfig.json	Adds reference to msgpack fs package.
pkg-manager/core/test/install/sideEffects.ts	Updates side-effects cache tests to use `.mpk` and `Map`.
pkg-manager/core/test/install/patch.ts	Updates patch tests to read `.mpk` index files and use `Map`.
pkg-manager/core/test/install/lockfileOnly.ts	Updates meta cache filename expectations to `.mpk`.
pkg-manager/core/package.json	Adds msgpack fs dependency.
modules-mounter/daemon/tsconfig.json	Adds reference to msgpack fs package.
modules-mounter/daemon/src/createFuseHandlers.ts	Reads `.mpk` indexes and uses `Map.get()` for file lookup.
modules-mounter/daemon/src/cafsExplorer.ts	Updates directory exploration helpers for `Map`-based files.
modules-mounter/daemon/package.json	Adds msgpack fs dep, removes `load-json-file`.
fs/msgpack-file/tsconfig.lint.json	Adds lint TS config for new package.
fs/msgpack-file/tsconfig.json	Adds build TS config for new package.
fs/msgpack-file/test/tsconfig.json	Adds test TS config for new package.
fs/msgpack-file/test/index.test.ts	Adds unit tests for msgpack read/write and Map/Set support.
fs/msgpack-file/src/index.ts	Implements msgpack read/write helpers using `msgpackr`.
fs/msgpack-file/package.json	Adds new workspace package definition.
fs/msgpack-file/README.md	Documents new msgpack fs utilities.
exec/plugin-commands-rebuild/tsconfig.json	Adds reference to msgpack fs package.
exec/plugin-commands-rebuild/test/index.ts	Updates rebuild tests for `.mpk` + `Map` sideEffects/files.
exec/plugin-commands-rebuild/src/implementation/index.ts	Reads `.mpk` indexes and checks sideEffects via `Map`.
exec/plugin-commands-rebuild/package.json	Adds msgpack fs dep, removes `load-json-file`.
cspell.json	Adds msgpack-related words to dictionary.
cache/commands/test/cacheList.cmd.test.ts	Updates cache list command tests for `.mpk` filenames.
cache/commands/test/cacheDelete.cmd.test.ts	Updates cache delete command tests for `.mpk` filenames.
cache/api/tsconfig.json	Adds reference to msgpack fs package.
cache/api/src/cacheView.ts	Reads cached meta from `.mpk` and updates glob patterns.
cache/api/src/cacheList.ts	Updates metadata file patterns to `.mpk`.
cache/api/package.json	Adds msgpack fs dep, removes `load-json-file`.
.changeset/msgpack-store-format.md	Declares major change: store/cache format switches to MessagePack.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

resolving/npm-resolver/test/utils/index.ts

worker/src/start.ts

Copilot

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

worker/src/start.ts

pnpm-workspace.yaml

Copilot

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg-manager/package-requester/test/index.ts

gluxon · 2026-01-22T23:52:05Z

Seems a bit faster! I'm running:

❯ hyperfine \
  --warmup 1 \
  --prepare "git restore pnpm-lock.yaml" \
  "pnpm install --ignore-scripts"

Commit d85ea8d

  Time (mean ± σ):     42.095 s ±  0.946 s    [User: 49.825 s, System: 26.705 s]
  Range (min … max):   40.545 s … 43.731 s    10 runs

Commit 3e7bebd

  Time (mean ± σ):     41.768 s ±  0.741 s    [User: 49.584 s, System: 26.453 s]
  Range (min … max):   40.432 s … 42.981 s    10 runs

Checking the CPU profile of these runs, most of the time is spent resolving dependencies and peers. Very little of the ~40s is spent deserializing MessagePack / JSON, so a pnpm install with a hot cache and store like the test I'm running above might not be the best for showcasing the performance improvement. It is still useful to know there isn't a performance regression though (which was surprisingly the case for the v8 buffers change).

zkochan · 2026-01-22T23:55:38Z

Don't you test with an up-to-date lockfile? In that case resolution will be skipped. Otherwise it will be hard to see the difference.

Alternatively, if you want to test with resolution, you can try --offline. In that case it will read the metadata .mpk file from cache.

zkochan · 2026-01-23T00:02:46Z

In any case, it is a bit faster according to your benchmarks. Thanks for the confirmation!

gluxon · 2026-01-23T19:23:44Z

Don't you test with an up-to-date lockfile? In that case resolution will be skipped. Otherwise it will be hard to see the difference.

I did notice your benchmarks tested with --frozen-lockfile. Please let me know if I'm missing something, but I was testing resolution intentionally because looking up index files and cached package metadata is dependent on the format these files are in? I was thinking we'd notice a greater differential when including resolveAndFetch calls in the benchmark.

Skipping resolution, we wouldn't benchmark deserialization of large cached package metadata.

Alternatively, if you want to test with resolution, you can try --offline. In that case it will read the metadata .mpk file from cache.

I should have benchmarked with --offline, you're right. I was expecting pnpm to realize cached metadata files were up to date and skip a full network request, but --offline would better ensure that.

I still need a lockfile that's not up-to-date for this to force resolution and reads of .mpk files from cache, even when passing --offline though, right?

gluxon · 2026-01-23T19:24:03Z

In any case, it is a bit faster according to your benchmarks. Thanks for the confirmation!

Indeed. Thank you for this performance improvement! Always happy to help.

zkochan · 2026-01-24T10:28:29Z

I am doing some more changes to the format but it doesn't seem to make a difference in performance: #10504

zkochan mentioned this pull request Jan 21, 2026

Performance regression after switching to v8 serialization format #10409

Closed

zkochan added the area: performance label Jan 22, 2026

zkochan added this to the v11.0 milestone Jan 22, 2026

perf: migrate internal cache and index files to MessagePack serializa…

f7fe178

…tion

zkochan force-pushed the perf/msgpackr branch from 81bc394 to f7fe178 Compare January 22, 2026 18:12

refactor: rename msgpack-serializer to fs.msgpack-file

c695e52

docs: add changeset

b2c4216

zkochan added 2 commits January 22, 2026 23:45

docs: add readme

e072da6

Merge remote-tracking branch 'origin/main' into perf/msgpackr

3f1ce53

zkochan marked this pull request as ready for review January 22, 2026 22:52

zkochan requested a review from weyert as a code owner January 22, 2026 22:52

Copilot AI review requested due to automatic review settings January 22, 2026 22:52

zkochan requested a review from weyert as a code owner January 22, 2026 22:52

Copilot started reviewing on behalf of zkochan January 22, 2026 22:52 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

resolving/npm-resolver/test/utils/index.ts Show resolved Hide resolved

worker/src/start.ts Outdated Show resolved Hide resolved

worker/src/start.ts Outdated Show resolved Hide resolved

fix: implemented CR suggestions

3e7bebd

zkochan requested a review from Copilot January 22, 2026 23:28

Copilot started reviewing on behalf of zkochan January 22, 2026 23:28 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

worker/src/start.ts Show resolved Hide resolved

pnpm-workspace.yaml Outdated Show resolved Hide resolved

fix: implemented CR suggestions

6c0749b

zkochan requested a review from Copilot January 22, 2026 23:43

Copilot started reviewing on behalf of zkochan January 22, 2026 23:43 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

pkg-manager/package-requester/test/index.ts Show resolved Hide resolved

zkochan merged commit 40b107e into main Jan 23, 2026
20 of 21 checks passed

zkochan deleted the perf/msgpackr branch January 23, 2026 00:31

Uh oh!

Conversation

zkochan commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Jan 22, 2026

Uh oh!

gluxon commented Jan 22, 2026

Uh oh!

zkochan commented Jan 22, 2026

Uh oh!

gluxon commented Jan 22, 2026

Benchmarking MessagePack

MessagePack (e841247)

main (c494de3)

Uh oh!

zkochan commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Jan 22, 2026

Uh oh!

gluxon commented Jan 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

gluxon commented Jan 22, 2026

Uh oh!

zkochan commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

gluxon commented Jan 23, 2026

Uh oh!

gluxon commented Jan 23, 2026

Uh oh!

zkochan commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zkochan commented Jan 21, 2026 •

edited

Loading

MessagePack (`e841247`)

main (`c494de3`)

zkochan commented Jan 22, 2026 •

edited

Loading

zkochan commented Jan 22, 2026 •

edited

Loading

zkochan commented Jan 23, 2026 •

edited

Loading