Describe the bug

There are some issues with the new bench function:

• several bench tests are shared between single suite, even if they are inside different describes:

describe('sort 3', () => {
  bench('normal 1', () => {
    const x = [1, 5, 4, 2, 3, 6];
    x.sort((a, b) => {
      return a - b;
    });
  });
});

describe('sort 2', () => {
  bench('normal 2', () => {
    const x = [1, 5, 4, 2, 3, 6];
    x.sort((a, b) => {
      return a - b;
    });
  });
});

Result:

 ✓ test/basic.bench.ts (2) 557ms
   ✓ sort 3 (1) 556ms
     · normal 1  515.09 ops/sec ±4.38% (257595 samples) fastest
   ✓ sort 2 (1) 556ms
     · normal 2  514.57 ops/sec ±5.11% (257595 samples) fastest

Single bench gives different results:

bench('normal 1', () => {
  const x = [1, 5, 4, 2, 3, 6];
  x.sort((a, b) => {
    return a - b;
  });
});

Result:

 ✓ test/basic.bench.ts (1) 557ms
   · normal 1  1,117.25 ops/sec ±4.17% (558625 samples) fastest

• options don't have any effect on benchmark

  bench(
    'normal 1',
    () => {
      const x = [1, 5, 4, 2, 3, 6];
      x.sort((a, b) => {
        return a - b;
      });
    },
    { time: 10000000 }
  );

bench/describe also doesn't support some API:

• bench.only (describe.only)
• bench.todo (describe.todo)

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-w6dzu4?file=test%2Fbasic.bench.ts

System Info

stackblitz

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.