-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Description
Describe the bug
vitest fails a shard if the shard index is higher than the number of test files.
For example, let's say there is only a single test file but you instruct vitest to split up
the test suite into two shards. While the first shard will pass, the second shard will fail
claiming that there are no test files found.
❯ npm run test:shard-2
> test:shard-2
> vitest run --shard=2/2
RUN v3.2.1 /Users/niklaas/git/red-guava/mre-vitest-shard-missing-files
No test files found, exiting with code 1
include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude: **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*
You can add the --passWithNotTests flag. However, this could cause you to miss out on no tests
being executed for the first shard if you execute vitest in a GitHub Workflow like
run: |
npm run test -- \
--shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
--passWithNoTestsfor example.
I think it's debatable whether this is an actual bug or whether the error message/behavior should just be improved. The case I'm describing is also pretty edgy: Why would you want to split your test suite into shards if you don't have that many test files?
To better explain why I ran into this: In the project I work on we run vitest twice with different TZ env variables set. Each run targets different test files, *.tz.test.js vs *.test.js. There's actually only a single *.tz.test.js file, but there are hundreds of *.test.js files, which is why we want to shard them.
Both executions of vitest we run in the same GitHub Workflow Job in two separate steps. It looks something like this:
- name: Run tests with custom timezone
id: js-unit-tz-tests
run: |
npm run test:timezone -- \
--retry=1 \
--shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
- name: Run tests without a custom timezone
id: js-unit-tests
run: |
npm run test -- \
--retry=1 \
--shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}And in the package.json scripts we have
"test": "NODE_ENV=test TZ=UTC vitest run",
"test:timezone": "NODE_ENV=test TZ='Australia/Melbourne' vitest run --config=vitest.tz.config.mjs",The vitest.tz.config.mjs configures the custom match on *.tz.test.js files.
Arguably, because there's just a single *.tz.test.js file, there is no need to shard it. But removing the --shard flag will cause it to run matrix.shardTotal number of times. Moving it to a separate job would require us to spin up and bootstrap a second runner.
The above got me questioning whether the error message No test files found, exiting with code 1 for a shard that "ran out of" test files is actually accurate, and whether the behavior to fail the execution is accurate. Arguably, you'd only want the execution(s) to fail if there aren't any test files for any shard, and the executions that don't have a test file left could print a warning saying something like No test files left for this sharded execution.
Again, this is pretty edgy, but it might be worth noting.
Sidenote: In version v3.1.4 our set-up above worked flawlessly because, while the error message claimed that it would exit with code 1, it actually exited with code 0:
❯ npm run test:shard-2
> test:shard-2
> vitest run --shard=2/2
RUN v3.1.4 /Users/niklaas/git/red-guava/mre-vitest-shard-missing-files
No test files found, exiting with code 1
include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude: **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*
❯ echo $?
0
Reproduction
https://github.com/redguava/mre-vitest-shard-missing-files
System Info
❯ npx envinfo --system --npmPackages '{vitest*,@vitest/*,vite,@vitejs/*,playwright,webdriverio}' --binaries --browsers
Need to install the following packages:
envinfo@7.14.0
Ok to proceed? (y) y
System:
OS: macOS 15.5
CPU: (16) arm64 Apple M3 Max
Memory: 2.34 GB / 64.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 22.13.0 - ~/.asdf/installs/nodejs/22.13.0/bin/node
npm: 10.9.2 - ~/.asdf/installs/nodejs/22.13.0/bin/npm
bun: 1.1.42 - /opt/homebrew/bin/bun
Browsers:
Safari: 18.5
npmPackages:
vitest: ^3.2.1 => 3.2.1Used Package Manager
npm
Validations
- Follow our Code of Conduct
- Read the Contributing Guidelines.
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
- The provided reproduction is a minimal reproducible example of the bug.