Skip to content

Fails a shard if the shard index is higher than the number of test files #8100

@niklaas

Description

@niklaas

Describe the bug

vitest fails a shard if the shard index is higher than the number of test files.

For example, let's say there is only a single test file but you instruct vitest to split up
the test suite into two shards. While the first shard will pass, the second shard will fail
claiming that there are no test files found.

❯ npm run test:shard-2

> test:shard-2
> vitest run --shard=2/2


 RUN  v3.2.1 /Users/niklaas/git/red-guava/mre-vitest-shard-missing-files

No test files found, exiting with code 1

include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude:  **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*

You can add the --passWithNotTests flag. However, this could cause you to miss out on no tests
being executed for the first shard if you execute vitest in a GitHub Workflow like

run: |
  npm run test -- \
  --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
  --passWithNoTests

for example.

I think it's debatable whether this is an actual bug or whether the error message/behavior should just be improved. The case I'm describing is also pretty edgy: Why would you want to split your test suite into shards if you don't have that many test files?

To better explain why I ran into this: In the project I work on we run vitest twice with different TZ env variables set. Each run targets different test files, *.tz.test.js vs *.test.js. There's actually only a single *.tz.test.js file, but there are hundreds of *.test.js files, which is why we want to shard them.

Both executions of vitest we run in the same GitHub Workflow Job in two separate steps. It looks something like this:

- name: Run tests with custom timezone
  id: js-unit-tz-tests
  run: |
    npm run test:timezone -- \
    --retry=1 \
    --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

- name: Run tests without a custom timezone
  id: js-unit-tests
  run: |
    npm run test -- \
    --retry=1 \
    --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

And in the package.json scripts we have

"test": "NODE_ENV=test TZ=UTC vitest run",
"test:timezone": "NODE_ENV=test TZ='Australia/Melbourne' vitest run --config=vitest.tz.config.mjs",

The vitest.tz.config.mjs configures the custom match on *.tz.test.js files.

Arguably, because there's just a single *.tz.test.js file, there is no need to shard it. But removing the --shard flag will cause it to run matrix.shardTotal number of times. Moving it to a separate job would require us to spin up and bootstrap a second runner.

The above got me questioning whether the error message No test files found, exiting with code 1 for a shard that "ran out of" test files is actually accurate, and whether the behavior to fail the execution is accurate. Arguably, you'd only want the execution(s) to fail if there aren't any test files for any shard, and the executions that don't have a test file left could print a warning saying something like No test files left for this sharded execution.

Again, this is pretty edgy, but it might be worth noting.

Sidenote: In version v3.1.4 our set-up above worked flawlessly because, while the error message claimed that it would exit with code 1, it actually exited with code 0:

❯ npm run test:shard-2

> test:shard-2
> vitest run --shard=2/2


 RUN  v3.1.4 /Users/niklaas/git/red-guava/mre-vitest-shard-missing-files

No test files found, exiting with code 1

include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude:  **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*


❯ echo $?
0

Reproduction

https://github.com/redguava/mre-vitest-shard-missing-files

System Info

❯ npx envinfo --system --npmPackages '{vitest*,@vitest/*,vite,@vitejs/*,playwright,webdriverio}' --binaries --browsers
Need to install the following packages:
envinfo@7.14.0
Ok to proceed? (y) y


  System:
    OS: macOS 15.5
    CPU: (16) arm64 Apple M3 Max
    Memory: 2.34 GB / 64.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.13.0 - ~/.asdf/installs/nodejs/22.13.0/bin/node
    npm: 10.9.2 - ~/.asdf/installs/nodejs/22.13.0/bin/npm
    bun: 1.1.42 - /opt/homebrew/bin/bun
  Browsers:
    Safari: 18.5
  npmPackages:
    vitest: ^3.2.1 => 3.2.1

Used Package Manager

npm

Validations

Metadata

Metadata

Assignees

No one assigned

    Labels

    p2-edge-caseBug, but has workaround or limited in scope (priority)

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions