Skip to content

sequence.shuffle: true does not respect the cache, runs test suites in random order #3568

@nstepien

Description

@nstepien

Describe the bug

I want tests to run in random order, and I also want slower test suites to run first and quicker test suites to run last, but it seems impossible without implementing a custom sequencer.

With test suites/files being isolated, I don't see the point of also running the test suites/files in random order.

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-c5wvvm?file=test%2Fsuite1.test.ts,test%2Fslow.test.ts,test%2Fsuite2.test.ts,vite.config.ts,test%2Fsuite0.test.ts&initialPath=__vitest__/

  1. run npm run test:run to build the cache
  2. try npm run test:run multiple times, you'll see the test suites run in random order
  3. set shuffle to false, run npm run test:run, notice that test/slow.test.ts starts first

System Info

System:
    OS: Windows 10 10.0.22621
    CPU: (64) x64 AMD Ryzen Threadripper 3970X 32-Core Processor
    Memory: 17.54 GB / 31.86 GB
  Binaries:
    Node: 20.2.0 - C:\Program Files\nodejs\node.EXE
    npm: 9.6.7 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.22621.1778.0), Chromium (114.0.1823.43)
    Internet Explorer: 11.0.22621.1
  npmPackages:
    @vitejs/plugin-react: ^4.0.0 => 4.0.0
    @vitest/coverage-v8: ^0.32.0 => 0.32.0
    vite: ^4.3.5 => 4.3.9
    vitest: ^0.32.0 => 0.32.0

Used Package Manager

npm

Validations

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions