feat: use SQLite for storing package index in the content-addressable store by zkochan · Pull Request #10827 · pnpm/pnpm

zkochan · 2026-03-01T23:38:35Z

Summary

Replace individual .mpk (MessagePack) files under $STORE/index/ with a single SQLite database at $STORE/index.db using Node.js 22's built-in node:sqlite module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.

Closes #10826

Design

New package: `@pnpm/store.index`

A new StoreIndex class wraps a SQLite database with a simple key-value API (get, set, delete, has, entries). Data is serialized with msgpackr and stored as BLOBs. The table uses WITHOUT ROWID for compact storage.

Key design decisions:

WAL mode enables concurrent reads from workers while the main process writes.
busy_timeout=5000 plus a retry loop with Atomics.wait-based sleepSync handles SQLITE_BUSY errors from concurrent access.
Performance PRAGMAs: synchronous=NORMAL, mmap_size=512MB, cache_size=32MB, temp_store=MEMORY, wal_autocheckpoint=10000.
Write batching: queueWrites() batches pre-packed entries from tarball extraction and flushes them in a single transaction on process.nextTick. setRawMany() writes immediate batches (e.g. from addFilesFromDir).
Lifecycle: close() auto-flushes pending writes, runs PRAGMA optimize, and closes the DB. A process.on('exit') handler ensures cleanup even on unexpected exits.
VACUUM after deleteMany (used by pnpm store prune) to reclaim disk space.

Key format

Keys are integrity\tpkgId (tab-separated). Git-hosted packages use pkgId\tbuilt or pkgId\tnot-built.

Shared StoreIndex instance

A single StoreIndex instance is threaded through the entire install lifecycle — from createNewStoreController through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.

Worker architecture

Index writes are performed in the main process, not in worker threads. Workers send pre-packed { key, buffer } pairs back to the main process via postMessage, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.

SQLite ExperimentalWarning suppression

node:sqlite emits an ExperimentalWarning on first load. This is suppressed via a process.emitWarning override injected through esbuild's banner option, which runs on line 1 of both dist/pnpm.mjs and dist/worker.js — before any module that loads node:sqlite.

No migration from `.mpk` files

Old .mpk index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).

Changed packages

121 files changed across these areas:

store/index/ — New @pnpm/store.index package
worker/ — Write batching moved from worker module into StoreIndex class; workers send pre-packed buffers to main process
store/package-store/ — StoreIndex creation and lifecycle management
store/cafs/ — Removed getFilePathInCafs index-file utilities (no longer needed)
store/pkg-finder/ — Reads from StoreIndex instead of .mpk files
store/plugin-commands-store/ — store status uses StoreIndex
store/plugin-commands-store-inspecting/ — cat-index and find-hash use StoreIndex
fetching/tarball-fetcher/ — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
fetching/git-fetcher/, binary-fetcher/, pick-fetcher/ — Accept StoreIndex parameter
pkg-manager/ — client, core, headless, package-requester thread StoreIndex
reviewing/ — license-scanner, sbom, dependencies-hierarchy accept StoreIndex
cache/api/ — Cache view uses StoreIndex
pnpm/bundle.ts — esbuild banner for ExperimentalWarning suppression

Test plan

pnpm --filter @pnpm/store.index test — Unit tests for StoreIndex CRUD and batching
pnpm --filter @pnpm/package-store test — Store controller lifecycle
pnpm --filter @pnpm/package-requester test — Package requester reads from SQLite index
pnpm --filter @pnpm/tarball-fetcher test — Tarball and git-hosted fetcher writes
pnpm --filter @pnpm/headless test — Headless install
pnpm --filter @pnpm/core test — Core install, side effects, patching
pnpm --filter @pnpm/plugin-commands-rebuild test — Rebuild reads from index
pnpm --filter @pnpm/license-scanner test — License scanning
e2e tests pass

🤖 Generated with Claude Code

… store Replace individual .mpk files under $STORE/index/ with a single SQLite database at $STORE/index.db using Node.js 22's built-in node:sqlite module. This reduces filesystem syscall overhead and improves space efficiency. SQLite WAL mode enables concurrent access across worker threads. Existing .mpk index files are transparently migrated to SQLite on read for backward compatibility. Closes #10826

github-actions · 2026-03-02T00:03:50Z

Benchmark Results

#	Scenario	main	branch
1	Headless (warm store+cache)	2.519s ± 0.078s	2.429s ± 0.053s
2	Re-resolution (add dep, warm)	3.974s ± 0.194s	3.978s ± 0.133s
3	Full resolution (warm, no lockfile)	7.650s ± 0.708s	7.339s ± 0.263s
4	Headless (cold store+cache)	6.393s ± 0.548s	6.643s ± 1.075s
5	Cold install (nothing warm)	11.674s ± 0.666s	11.336s ± 0.547s
6	GVS warm reinstall (warm global store)	1.402s ± 0.027s	1.402s ± 0.015s

Run 22743514397 · 30 runs per scenario · triggered by @zkochan

exec/plugin-commands-rebuild/src/implementation/index.ts

store/index/src/index.ts

…f filesystem paths Replace getIndexFilePathInCafs with storeIndexKey across the codebase. Instead of constructing full filesystem paths (hex extraction, character escaping, path building), StoreIndex keys are now simply `${integrity}\t${pkgId}`. This removes the hex conversion, character escaping, and path construction from the hot path. - Remove getIndexFilePathInCafs and getIndexKeyInCafs from store/cafs - Remove @pnpm/crypto.integrity dependency from store/cafs (no longer needed) - Add storeIndexKey() and has() to @pnpm/store-index - Update all 20+ callers to use storeIndexKey from @pnpm/store-index Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zkochan · 2026-03-02T13:56:25Z

so far it seems like it is a bit faster with hot cache and a bit slower with cold cache.

Workers now pre-pack index data with msgpackr and return raw buffers. The main process writes them to SQLite via setRawMany(), avoiding cross-thread Packr instance incompatibility (useRecords=true causes Maps to deserialize as plain objects across different Packr instances). Tarball writes are batched via process.nextTick; side-effect writes are immediate so subsequent worker reads see committed data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Instead of letting structured clone copy the packed buffer, wrap it in a SharedArrayBuffer so it transfers to the main process without copying. This preserves the single-writer architecture while avoiding the overhead of buffer copies through postMessage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zkochan · 2026-03-02T15:26:45Z

Seems like it is faster if we write to the DB only in the main process. Reading in the workers is fine.

We also need to be sure that this approach will be faster when the store gets really big. 10th of thousands of packages.

Only setRawMany is used (called from the main process with pre-packed buffers from workers).

SukkaW · 2026-03-02T20:42:38Z

Also, we may speed up messagepack a bit (not actually related to SQLite though, but would be a nice to have).

I just found out msgpackr's Shared Record Structures feature, which provides getStructures and saveStructures, and is "robust and can be used in multiple-process situations".

We may persist this in SQLite as well since most of the data stored in StoreIndex would have an identical shape.

This technique may be preferred over a shared Packr instance (5617a32).

zkochan · 2026-03-02T21:00:48Z

I know about shared record structures but it makes little sense to use them for index files. The overhead of storing the structures by file is minimal and it makes the index files more resilient. Maybe it is different in case of sqlight

Use msgpackr shared record structures for data stored in SQLite, reducing serialized size by ~38% for package index entries. The common object shapes (PackageFilesIndex, PackageFileInfo, and BundledManifest fields) are pre-defined and persisted in a msgpack_structures table. Workers pack using the StoreIndex's Packr instance via SharedArrayBuffer. File-based .mpk entries continue to use plain msgpackr without shared structures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zkochan · 2026-03-03T00:11:43Z

~~It doesn't look like it makes a meaningful difference.~~

With shared structures it seems to be slower. I have reverted that change.

Summary from AI about the latest benchmark run

Benchmark	main	branch	delta	significant?
Headless (warm)	2.548s ± 0.096	2.354s ± 0.141	-7.6%	Yes — branch min (1.856s) well below main's range
Re-resolution (warm)	4.070s ± 0.391	3.942s ± 0.244	-3.1%	Marginal — overlapping σ bands
Full resolution (warm)	7.121s ± 0.188	7.427s ± 1.731	+4.3%	No — huge σ on branch, outlier to 16.5s
Headless (cold store)	6.428s ± 0.371	6.365s ± 0.355	-1.0%	No — within noise
Cold install	10.879s ± 0.271	11.021s ± 0.483	+1.3%	No — within noise

Benchmark 1 (the most common real-world scenario — frozen lockfile, warm store) shows a solid ~8% improvement. This makes sense: SQLite reads from a single indexed file are faster than thousands of individual readFileSync + directory traversal calls for .mpk files.

Benchmarks 4 and 5 (cold store, heavy writes) are essentially at parity, which means the single-writer batching compensates for the overhead of the worker→main process transfer.

Benchmark 3's high variance (σ of 1.7s, outlier at 16.5s) suggests something intermittent — possibly WAL checkpoint or GC pressure during that specific workload. Worth ignoring for now.

Overall: SQLite is a clear win for the warm/hot path that users hit most often, and neutral for cold installs.

This reverts commit 0ef2e71.

zkochan · 2026-03-03T01:10:25Z

@pnpm/collaborators thoughts?

setRawMany was unconditionally writing all entries to SQLite, but git-hosted packages use filesystem paths as keys (not tab-separated SQLite keys). This caused ENOENT when gitHostedTarballFetcher tried to rename the .mpk file that was never written to disk. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

store/store-index/src/index.ts

ryo-manba

Overall looks good! Really cool idea 😄

synchronous=OFF risks database corruption on power loss. NORMAL is safe in WAL mode and still performant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

This PR replaces the pnpm content-addressable store’s per-package index files with a single SQLite database ($STORE/index.db) via a new @pnpm/store.index package, and threads a StoreIndex instance through the fetch/store/worker pipeline to reduce filesystem overhead.

Changes:

Introduces @pnpm/store.index (SQLite + msgpackr) and updates store index keying to use integrity\tpkgId.
Refactors store reads/writes to use SQLite-backed StoreIndex, including batching index writes in the main process and keeping worker threads read-only.
Updates multiple commands/tests/utilities to use StoreIndex and closes SQLite handles during teardown.

Reviewed changes

Copilot reviewed 119 out of 121 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
worker/tsconfig.json	Updates TS project references for new store index package.
worker/src/worker.ts	Adds warning handler (but currently too late due to ESM import order).
worker/src/start.ts	Worker-side: replaces msgpack file reads/writes with SQLite index reads and returns batched index writes to main thread.
worker/src/index.ts	Main-process batching/flushing of SQLite index writes; defers DB close until workers finish.
worker/package.json	Replaces `@pnpm/fs.msgpack-file` dependency with `@pnpm/store.index`.
testing/temp-store/tsconfig.json	Adds TS reference to `store/index`.
testing/temp-store/src/index.ts	Creates/passes `StoreIndex` in temp store helper.
testing/temp-store/package.json	Adds `@pnpm/store.index` dependency.
store/store-connection-manager/tsconfig.json	Adds TS reference to `store/index`.
store/store-connection-manager/src/createNewStoreController.ts	Creates and threads `StoreIndex` through client and package store creation.
store/store-connection-manager/package.json	Adds `@pnpm/store.index` dependency.
store/plugin-commands-store/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
store/plugin-commands-store/test/storeStatus.ts	Stabilizes subprocess config via clean config dir.
store/plugin-commands-store/src/storeStatus/index.ts	Reads package indexes via `StoreIndex` using new key helpers; ensures close in finally.
store/plugin-commands-store/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
store/plugin-commands-store-inspecting/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
store/plugin-commands-store-inspecting/test/findHash.ts	Adjusts expectations for new SQLite index key output.
store/plugin-commands-store-inspecting/src/findHash.ts	Iterates SQLite index entries instead of scanning index directory.
store/plugin-commands-store-inspecting/src/catIndex.ts	Reads index entry via `StoreIndex`; throws when missing; closes in finally.
store/plugin-commands-store-inspecting/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
store/pkg-finder/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
store/pkg-finder/src/index.ts	Reads package file map via `StoreIndex`; throws ENOENT when missing.
store/pkg-finder/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
store/package-store/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
store/package-store/test/index.ts	Passes `StoreIndex` into client/package store in tests.
store/package-store/src/storeController/prune.ts	Prunes orphaned index entries via `storeIndex.entries()`/`deleteMany()` instead of deleting `.mpk` files.
store/package-store/src/storeController/index.ts	Requires `storeIndex` in options; flushes/defer-closes index on store close; passes storeIndex to worker upload.
store/package-store/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
store/index/tsconfig.lint.json	New lint tsconfig for `@pnpm/store.index`.
store/index/tsconfig.json	New TS project for `@pnpm/store.index`.
store/index/test/tsconfig.json	New test TS config for store index package.
store/index/test/index.ts	Adds basic StoreIndex round-trip + entries() tests.
store/index/src/index.ts	Implements SQLite-backed StoreIndex with WAL tuning and msgpackr packing.
store/index/package.json	New package definition for `@pnpm/store.index` (Node >=22.13).
store/index/README.md	Documents rationale for moving to SQLite-backed store index.
store/cafs/tsconfig.json	Removes unused TS reference to crypto integrity module.
store/cafs/src/index.ts	Removes `getIndexFilePathInCafs` from CAFS public surface.
store/cafs/src/getFilePathInCafs.ts	Removes `getIndexFilePathInCafs` implementation (index is now SQLite).
store/cafs/package.json	Drops `@pnpm/crypto.integrity` dependency (no longer used for index paths).
store/cafs-types/src/index.ts	Removes `getIndexFilePathInCafs` from CAFS types.
reviewing/sbom/tsconfig.json	Adds TS reference to `store/index`.
reviewing/sbom/src/getPkgMetadata.ts	Adds `storeIndex` to metadata options shape.
reviewing/sbom/src/collectComponents.ts	Creates and passes StoreIndex for metadata reads; closes after walk (currently not in finally).
reviewing/sbom/package.json	Adds `@pnpm/store.index` dependency.
reviewing/outdated/src/createManifestGetter.ts	Updates ClientOptions omission to exclude new `storeIndex` field.
reviewing/license-scanner/tsconfig.json	Adds TS reference to `store/index`.
reviewing/license-scanner/test/licenses.spec.ts	Uses a real temp storeDir instead of hardcoded `/opt/.pnpm`.
reviewing/license-scanner/test/getPkgInfo.spec.ts	Creates/closes StoreIndex in tests; updates error regex.
reviewing/license-scanner/src/lockfileToLicenseNodeTree.ts	Threads StoreIndex into getPkgInfo path; closes at end (currently not in finally).
reviewing/license-scanner/src/getPkgInfo.ts	Requires `storeIndex` in options and passes it into pkg-finder.
reviewing/license-scanner/package.json	Adds `@pnpm/store.index` dependency.
reviewing/dependencies-hierarchy/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
reviewing/dependencies-hierarchy/src/readManifestFromCafs.ts	Reads manifest via `StoreIndex` key instead of CAFS index file path.
reviewing/dependencies-hierarchy/src/getTree.ts	Adds optional `storeIndex` plumbing to tree opts.
reviewing/dependencies-hierarchy/src/getPkgInfo.ts	Uses storeIndex-backed manifest peek when available.
reviewing/dependencies-hierarchy/src/buildDependentsTree.ts	Opens StoreIndex for store reads; closes at end.
reviewing/dependencies-hierarchy/src/buildDependenciesTree.ts	Opens StoreIndex for store reads; closes at end (currently not in finally).
reviewing/dependencies-hierarchy/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
resolving/npm-resolver/tsconfig.json	Adds TS reference to `store/index`.
resolving/npm-resolver/src/index.ts	Computes index key via `storeIndexKey` instead of CAFS index file path.
resolving/npm-resolver/package.json	Adds `@pnpm/store.index` dependency.
pnpm/tsconfig.json	Adds TS reference to `store/index`.
pnpm/test/install/misc.ts	Updates tests to read/write store index via StoreIndex instead of `.mpk` files.
pnpm/package.json	Adds `@pnpm/store.index` dependency.
pnpm/bin/pnpm.mjs	Adds warning handler (but currently too late due to ESM import order).
pnpm-lock.yaml	Lockfile updates to reflect dependency graph changes (including new store/index importer).
pkg-manager/plugin-commands-installation/tsconfig.json	Adds TS reference to `worker`.
pkg-manager/plugin-commands-installation/test/fetch.ts	Ensures workers/SQLite handles are closed before deleting store directory.
pkg-manager/plugin-commands-installation/package.json	Adds `@pnpm/worker` dev dependency for tests.
pkg-manager/package-requester/tsconfig.json	Adds TS reference to `store/index`.
pkg-manager/package-requester/test/index.ts	Updates tests to pass StoreIndex and to read index entries via StoreIndex.
pkg-manager/package-requester/src/packageRequester.ts	Uses `storeIndexKey`/`gitHostedStoreIndexKey` instead of CAFS index file path function.
pkg-manager/package-requester/package.json	Adds `@pnpm/store.index` dependency.
pkg-manager/headless/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
pkg-manager/headless/test/index.ts	Updates tests to edit/read index entries via StoreIndex.
pkg-manager/headless/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
pkg-manager/core/tsconfig.json	Adds TS reference to `store/index`.
pkg-manager/core/test/install/sideEffects.ts	Updates side-effects-cache tests to use StoreIndex.
pkg-manager/core/test/install/patch.ts	Updates patch tests to use StoreIndex and registry mock integrity helper.
pkg-manager/core/package.json	Adds `@pnpm/store.index` dependency.
pkg-manager/client/tsconfig.json	Adds TS reference to `store/index`.
pkg-manager/client/test/index.ts	Updates tests to provide and close StoreIndex.
pkg-manager/client/src/index.ts	Makes `storeIndex` a required ClientOptions field; narrows createResolver signature.
pkg-manager/client/package.json	Adds `@pnpm/store.index` dependency.
modules-mounter/daemon/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
modules-mounter/daemon/src/createFuseHandlers.ts	Switches to StoreIndex for index reads (currently opens/closes per lookup).
modules-mounter/daemon/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
fetching/tarball-fetcher/tsconfig.json	Adds TS reference to `store/index`.
fetching/tarball-fetcher/test/fetch.ts	Creates/closes StoreIndex used by tarball fetchers in tests.
fetching/tarball-fetcher/src/remoteTarballFetcher.ts	Passes StoreIndex through remote tarball add-to-store path.
fetching/tarball-fetcher/src/localTarballFetcher.ts	Requires StoreIndex in local tarball fetcher factory and passes to worker.
fetching/tarball-fetcher/src/index.ts	Threads StoreIndex into tarball fetching pipeline and fetchFromTarball context.
fetching/tarball-fetcher/src/gitHostedTarballFetcher.ts	Replaces temp index file rename with raw/final StoreIndex keys and DB operations.
fetching/tarball-fetcher/package.json	Adds `@pnpm/store.index`; removes `path-temp`/`rename-overwrite` usage here.
fetching/pick-fetcher/tsconfig.json	Adds TS reference to `store/index`.
fetching/pick-fetcher/test/customFetch.ts	Updates tarball fetcher construction to include StoreIndex.
fetching/pick-fetcher/package.json	Adds `@pnpm/store.index` dependency.
fetching/git-fetcher/tsconfig.json	Adds TS reference to `store/index`.
fetching/git-fetcher/test/index.ts	Updates git fetcher tests to provide and close StoreIndex.
fetching/git-fetcher/src/index.ts	Requires StoreIndex in options and passes to worker addFilesFromDir.
fetching/git-fetcher/package.json	Adds `@pnpm/store.index` dependency.
fetching/binary-fetcher/tsconfig.json	Adds TS reference to `store/index`.
fetching/binary-fetcher/src/index.ts	Threads StoreIndex into binary fetcher addFilesFromDir calls.
fetching/binary-fetcher/package.json	Adds `@pnpm/store.index` dependency.
exec/plugin-commands-rebuild/tsconfig.json	Adds TS reference to `store/index` and removes msgpack-file reference.
exec/plugin-commands-rebuild/test/index.ts	Updates tests to read/write store index entries via StoreIndex.
exec/plugin-commands-rebuild/src/implementation/index.ts	Uses StoreIndex for side-effects cache checks/paths and closes it at end.
exec/plugin-commands-rebuild/package.json	Adds `@pnpm/store.index` and removes `@pnpm/fs.msgpack-file`.
env/node.fetcher/tsconfig.json	Adds TS reference to `store/index`.
env/node.fetcher/test/node.test.ts	Updates tests to provide and close StoreIndex.
env/node.fetcher/src/index.ts	Requires StoreIndex in options; passes it into tarball fetcher creation.
env/node.fetcher/package.json	Adds `@pnpm/store.index` dependency.
cspell.json	Adds new SQLite-related terms to dictionary.
cache/api/tsconfig.json	Adds TS reference to `store/index`.
cache/api/src/cacheView.ts	Uses StoreIndex.has() to determine cached versions.
cache/api/package.json	Adds `@pnpm/store.index` dependency.
utils/jest-config/with-registry/jest-preset.js	Forces Jest to exit after teardown (mitigates leaked handles).
utils/assert-store/tsconfig.json	Adds TS reference to `store/index`.
utils/assert-store/src/index.ts	Updates store assertions to check index entries via StoreIndex.
utils/assert-store/package.json	Adds `@pnpm/store.index` dependency.
.changeset/sqlite-store-index.md	Adds changeset describing move to SQLite and (currently) claims legacy migration.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.changeset/sqlite-store-index.md

store/index/src/index.ts

modules-mounter/daemon/src/createFuseHandlers.ts

reviewing/sbom/src/collectComponents.ts

reviewing/license-scanner/src/lockfileToLicenseNodeTree.ts

reviewing/dependencies-hierarchy/src/buildDependenciesTree.ts

worker/src/worker.ts

…ctly StoreIndex.queueWrites() and flush() replace the module-level batching in the worker package. close() now auto-flushes, so the deferred-close pattern (deferStoreIndexClose/closeStoreIndexes/flushStoreIndexWrites) is no longer needed — the store controller closes the index directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

PRAGMA optimize and db.close() can throw if another connection holds the lock. Neither is critical — data is already flushed, and the OS reclaims the connection on process exit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Auto-close StoreIndex on process exit via exit handler, so callers that forget to call close() don't leak SQLite handles - Reuse single StoreIndex in FUSE daemon instead of open/close per cache miss - Replace process.removeAllListeners('warning') with targeted process.emit override that only suppresses SQLite ExperimentalWarning - Fix changeset: no transparent migration exists, packages are re-fetched on demand Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

storeController.close() can be called multiple times (headlessInstall and mutateModules finally both call it). Closing the DB on the first call causes "statement has been finalized" errors on subsequent reads. Instead, just flush pending writes — the process exit handler in the StoreIndex constructor handles the actual DB close on shutdown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

finishWorkers() now calls closeAllStoreIndexes() to release SQLite DB file handles, preventing EBUSY errors on Windows when tests remove the store directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…lose from headlessInstall storeController.close() now calls storeIndex.close() directly. Removed the redundant storeController.close() from headlessInstall since mutateModules already closes it in its finally block. This eliminates the double-close that caused "statement has been finalized" errors with git-hosted packages. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

store/package-store/src/storeController/prune.ts

storeController.close() now only flushes pending writes — closing the SQLite DB is left to the process exit handler. This prevents "statement has been finalized" errors when git-hosted package fetches outlive the mutateModules finally block. Tests that need to rimraf the store call closeAllStoreIndexes() directly after finishWorkers(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 120 out of 122 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

store/index/src/index.ts

fetching/tarball-fetcher/src/gitHostedTarballFetcher.ts

Copilot · 2026-03-06T09:07:32Z

worker/src/worker.ts

+// Suppress "SQLite is an experimental feature" warnings without
+// removing other warning listeners.
+const originalEmit = process.emit.bind(process) as typeof process.emit
+process.emit = function (event: string, ...args: unknown[]) {
+  if (event === 'warning' && args[0] instanceof Error &&
+      args[0].name === 'ExperimentalWarning' &&
+      args[0].message.includes('SQLite')) {
+    return false
+  }
+  return (originalEmit as Function).call(process, event, ...args) // eslint-disable-line @typescript-eslint/no-unsafe-function-type
+} as typeof process.emit


Overriding process.emit is very invasive and the current signature only accepts event: string, but EventEmitter.emit can be called with string | symbol. This risks breaking non-string process events and makes debugging harder. Prefer a narrower hook (e.g. monkeypatch process.emitWarning only) or a wrapper that preserves the original emit signature (string | symbol) and behavior as closely as possible.

Reclaim disk space after bulk deletions (e.g. pnpm store prune) by running VACUUM inside deleteMany. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move the SQLite ExperimentalWarning suppression from inline code in pnpm/bin/pnpm.mjs and worker/src/worker.ts into the esbuild banner in bundle.ts. The banner runs on line 1 of both dist/pnpm.mjs and dist/worker.js, before any module code that loads node:sqlite. The previous approach (inline code in entry modules) didn't work because ESM hoists static imports before module body code, so node:sqlite was loaded before the suppression was set up. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… sleep buffer - Flush queued store index writes before reading the raw files index in the git-hosted tarball fetcher to avoid a race where the data is still in pendingWrites and not yet committed to SQLite. - Reuse a static Int32Array for sleepSync to avoid allocating a new SharedArrayBuffer on every retry. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

This reverts commit 5a96bf1.

…warning suppression Override process.emitWarning instead of the more invasive process.emit to suppress SQLite ExperimentalWarning. This is narrower in scope and avoids interfering with the full event emitter signature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zkochan · 2026-03-06T10:37:53Z

I hope it won't have to be reverted. I'll merge and release it to be able to test more thoroughly.

…currency Move busy_timeout=5000 to be the first PRAGMA executed, before journal_mode=WAL and CREATE TABLE. Without the busy handler active, concurrent processes (e.g. parallel dlx calls sharing the same store) could get an immediate SQLITE_BUSY on Windows where file locking is mandatory. Also wrap all initialization PRAGMAs in a single sqliteRetry block so the entire sequence is retried on contention. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ved getIndexFilePathInCafs The store index was migrated from msgpack files to SQLite in #10827, but this test still referenced the removed getIndexFilePathInCafs function and readMsgpackFileSync, causing a compilation error.

zkochan added 2 commits March 2, 2026 00:37

test: fix

905436d

zkochan mentioned this pull request Mar 2, 2026

[RFC] Use SQLite for package metadata within pnpm store #10826

Closed

1 task

perf: sqlight

5aec007

SukkaW reviewed Mar 2, 2026

View reviewed changes

exec/plugin-commands-rebuild/src/implementation/index.ts Outdated Show resolved Hide resolved

SukkaW reviewed Mar 2, 2026

View reviewed changes

store/index/src/index.ts Show resolved Hide resolved

zkochan and others added 2 commits March 2, 2026 13:43

perf: increase SQLite page cache and use in-memory temp store

ffebe3a

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zkochan and others added 2 commits March 2, 2026 15:18

zkochan added 2 commits March 2, 2026 16:43

refactor: remove unused setMany method from StoreIndex

302b28c

Only setRawMany is used (called from the main process with pre-packed buffers from workers).

perf: optimize SQLite store index operations

68c0af1

revert: "perf: add msgpackr shared structures for SQLite index entries"

f3cb4ee

This reverts commit 0ef2e71.

zkochan and others added 2 commits March 3, 2026 08:15

test: update patch tests to read index from SQLite

17faf5c

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ryo-manba reviewed Mar 3, 2026

View reviewed changes

store/store-index/src/index.ts Outdated Show resolved Hide resolved

ryo-manba reviewed Mar 3, 2026

View reviewed changes

zkochan and others added 5 commits March 3, 2026 19:32

fix: use synchronous=NORMAL instead of OFF for SQLite store index

356d3e7

synchronous=OFF risks database corruption on power loss. NORMAL is safe in WAL mode and still performant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

test: read package-requester index from SQLite instead of mpk file

a3732b8

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix: suppress SQLite ExperimentalWarning in CLI and worker

1827436

Merge branch 'main' into sqlight

dee211d

test: fix

8688c65

zkochan marked this pull request as ready for review March 5, 2026 23:51

zkochan requested review from gluxon and weyert as code owners March 5, 2026 23:51

Copilot AI review requested due to automatic review settings March 5, 2026 23:51

Copilot started reviewing on behalf of zkochan March 5, 2026 23:52 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

zkochan and others added 5 commits March 6, 2026 01:09

fix: close all StoreIndex instances when finishing workers

3ac17cc

finishWorkers() now calls closeAllStoreIndexes() to release SQLite DB file handles, preventing EBUSY errors on Windows when tests remove the store directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zkochan force-pushed the sqlight branch from 6d39dc8 to 3ac17cc Compare March 6, 2026 07:44

SukkaW reviewed Mar 6, 2026

View reviewed changes

store/package-store/src/storeController/prune.ts Show resolved Hide resolved

zkochan and others added 2 commits March 6, 2026 09:30

chore: update tsconfig.json

11a991e

zkochan requested a review from Copilot March 6, 2026 08:59

Copilot started reviewing on behalf of zkochan March 6, 2026 08:59 View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

zkochan and others added 6 commits March 6, 2026 10:08

feat: run VACUUM after deleteMany in StoreIndex

e63a98a

Reclaim disk space after bulk deletions (e.g. pnpm store prune) by running VACUUM inside deleteMany. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chore: remove forceExit

5a96bf1

revert: "chore: remove forceExit"

3313111

This reverts commit 5a96bf1.

zkochan merged commit b7f0f21 into main Mar 6, 2026
12 of 13 checks passed

zkochan deleted the sqlight branch March 6, 2026 11:59

Uh oh!

Conversation

zkochan commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

New package: @pnpm/store.index

Key format

Shared StoreIndex instance

Worker architecture

SQLite ExperimentalWarning suppression

No migration from .mpk files

Changed packages

Test plan

Uh oh!

github-actions bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Results

Uh oh!

Uh oh!

Uh oh!

zkochan commented Mar 2, 2026

Uh oh!

zkochan commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SukkaW commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Mar 2, 2026

Uh oh!

zkochan commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ryo-manba left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

zkochan commented Mar 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zkochan commented Mar 1, 2026 •

edited

Loading

New package: `@pnpm/store.index`

No migration from `.mpk` files

github-actions bot commented Mar 2, 2026 •

edited

Loading

zkochan commented Mar 2, 2026 •

edited

Loading

SukkaW commented Mar 2, 2026 •

edited

Loading

zkochan commented Mar 3, 2026 •

edited

Loading

zkochan commented Mar 3, 2026 •

edited

Loading