vitest-pool-runner - Concurrent DO requests that throw result in failure to pop isolated storage stack frame #11031
Description
What versions & operating system are you using?
System:
OS: Linux 6.14 Ubuntu 24.04.3 LTS 24.04.3 LTS (Noble Numbat)
CPU: (20) x64 13th Gen Intel(R) Core(TM) i7-13800H
Memory: 10.83 GB / 30.96 GB
Container: Yes
Shell: 5.2.21 - /bin/bash
Binaries:
Node: 23.11.1 - /home/matthewrodgers/.nvm/versions/node/v23.11.1/bin/node
npm: 10.9.2 - /home/matthewrodgers/.nvm/versions/node/v23.11.1/bin/npm
pnpm: 10.18.3 - /home/matthewrodgers/.local/share/pnpm/pnpm
bun: 1.2.20 - /home/matthewrodgers/.bun/bin/bun
Deno: 2.5.0 - /home/matthewrodgers/.deno/bin/deno
npmPackages:
@cloudflare/vitest-pool-workers: ^0.9.13 => 0.9.13
wrangler: ^4.43.0 => 4.43.0
Please provide a link to a minimal reproduction
https://github.com/matthewdavidrodgers/pop-do-storage-bug-repro
Describe the Bug
Concurrent requests to a Durable Object that throw an exception over RPC (even when caught by the binding Worker) cause an uncaught exception in the vitest-pool-runner, as the runner is unable to pop the isolated storage stack frame for the Durable Object. Specifically, a sqlite shared memory file is still present (even if there are no writes to storage), presumably from concurrent operation https://www.sqlite.org/tempfiles.html. Workerd documents that this may be possible https://github.com/cloudflare/workerd/blob/main/src/workerd/server/workerd.capnp#L697, but it seems there's some cleanup that fails or happens too late.
Note that in the reproduction shared, there's an additional skipped test that performs more work like mocking a KV binding and throwing from that binding. This was the original failure case that got me looking into this, but a more minimal reproduction just involves an RPC method in a DO that throws.
Please provide any relevant error logs
$ npm run test
> pop-do-storage-bug-repro@0.0.0 test
> vitest run
RUN v3.2.4 /home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro
[vpw:info] Starting isolated runtimes for vitest.config.mts...
[mf:warn] The latest compatibility date supported by the installed Cloudflare Workers Runtime is "2025-10-08",
but you've requested "2025-10-11". Falling back to "2025-10-08"...
workerd/io/worker.c++:2230: info: uncaught exception; source = Uncaught (in promise); stack = Error: an error
at MyDurableObject.justThrow (/home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/src/index.ts:13:9)
at DurableObject.fn (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-internal.mjs:358:14)
workerd/io/worker.c++:2230: info: uncaught exception; source = Uncaught (in promise); stack = Error: an error
at MyDurableObject.justThrow (/home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/src/index.ts:13:9)
at DurableObject.fn (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-internal.mjs:358:14)
================================================================================
Failed to pop isolated storage stack frame in test/index.spec.ts's test "should throw concurrently".
In particular, we were unable to pop Durable Objects storage.
This usually means your Worker tried to access storage outside of a test, or some resources have not been disposed of properly.
Ensure you "await" all Promises that read or write to these services, and make sure you use the "using" keyword when passing data across JSRPC.
See https://developers.cloudflare.com/workers/testing/vitest-integration/known-issues/#isolated-storage for more details.
- AssertionError [ERR_ASSERTION]: Expected .sqlite, got /tmp/miniflare-ec6a771be35596ce336d904730c5e8c7/do/vitest-pool-workers-runner--__VITEST_POOL_WORKERS_USER_OBJECTMyDurableObject/ca3d053ec276c29fded9eaae04d42236a20e511fe580adaee779ca591712b971.sqlite-shm
================================================================================
Error running worker: AssertionError [ERR_ASSERTION]: Isolated storage failed. There should be additional logs above.
at new AssertionError (node-internal:internal_assertionerror:427:15)
at assert.strictEqual (node-internal:internal_assert:196:11)
at WorkersTestRunner.updateStackedStorage (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:133:12)
at WorkersTestRunner.onAfterRunSuite (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:171:5)
at runSuite (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@vitest/runner/dist/chunk-hooks.js:1763:3)
at runFiles (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3)
at startTests (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3)
at home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
at withEnv (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3)
at run (home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2)
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: Isolated storage failed. There should be additional logs above.
❯ new AssertionError node-internal:internal_assertionerror:427:15
❯ assert.strictEqual node-internal:internal_assert:196:11
❯ WorkersTestRunner.updateStackedStorage home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:133:12
❯ WorkersTestRunner.onAfterRunSuite home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:171:5
❯ runSuite home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@vitest/runner/dist/chunk-hooks.js:1763:3
❯ runFiles home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3
❯ startTests home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3
❯ home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
❯ withEnv home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3
❯ run home/matthewrodgers/code/misc/workers/pop-do-storage-bug-repro/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Test Files (1)
Tests 1 skipped (2)
Errors 1 error
Start at 08:19:33
Duration 2.31s (transform 59ms, setup 0ms, collect 76ms, tests 0ms, environment 0ms, prepare 379ms)
[vpw:debug] Shutting down runtimes...
Metadata
Metadata
Assignees
Type
Projects
Status