Skip to content

Commit 9e4cfd2

Browse files
MazenSamehRmatanshavitAriPerkkiosheremet-va
authored
feat(runner): enhance retry options (#9370)
Co-authored-by: Matan Shavit <71092861+matanshavit@users.noreply.github.com> Co-authored-by: Ari Perkkiö <ari.perkkio@gmail.com> Co-authored-by: Vladimir <sleuths.slews0s@icloud.com>
1 parent f5e6866 commit 9e4cfd2

File tree

15 files changed

+482
-21
lines changed

15 files changed

+482
-21
lines changed

‎docs/config/retry.md‎

Lines changed: 136 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,141 @@ outline: deep
55

66
# retry
77

8-
- **Type:** `number`
8+
Retry the test specific number of times if it fails.
9+
10+
- **Type:** `number | { count?: number, delay?: number, condition?: RegExp }`
911
- **Default:** `0`
10-
- **CLI:** `--retry=<value>`
12+
- **CLI:** `--retry <times>`, `--retry.count <times>`, `--retry.delay <ms>`, `--retry.condition <pattern>`
1113

12-
Retry the test specific number of times if it fails.
14+
## Basic Usage
15+
16+
Specify a number to retry failed tests:
17+
18+
```ts
19+
export default defineConfig({
20+
test: {
21+
retry: 3,
22+
},
23+
})
24+
```
25+
26+
## CLI Usage
27+
28+
You can also configure retry options from the command line:
29+
30+
```bash
31+
# Simple retry count
32+
vitest --retry 3
33+
34+
# Advanced options using dot notation
35+
vitest --retry.count 3 --retry.delay 500 --retry.condition 'ECONNREFUSED|timeout'
36+
```
37+
38+
## Advanced Options <Version>4.1.0</Version> {#advanced-options}
39+
40+
Use an object to configure retry behavior:
41+
42+
```ts
43+
export default defineConfig({
44+
test: {
45+
retry: {
46+
count: 3, // Number of times to retry
47+
delay: 1000, // Delay in milliseconds between retries
48+
condition: /ECONNREFUSED|timeout/i, // RegExp to match errors that should trigger retry
49+
},
50+
},
51+
})
52+
```
53+
54+
### count
55+
56+
Number of times to retry a test if it fails. Default is `0`.
57+
58+
```ts
59+
export default defineConfig({
60+
test: {
61+
retry: {
62+
count: 2,
63+
},
64+
},
65+
})
66+
```
67+
68+
### delay
69+
70+
Delay in milliseconds between retry attempts. Useful for tests that interact with rate-limited APIs or need time to recover. Default is `0`.
71+
72+
```ts
73+
export default defineConfig({
74+
test: {
75+
retry: {
76+
count: 3,
77+
delay: 500, // Wait 500ms between retries
78+
},
79+
},
80+
})
81+
```
82+
83+
### condition
84+
85+
A RegExp pattern or a function to determine if a test should be retried based on the error.
86+
87+
- When a **RegExp**, it's tested against the error message
88+
- When a **function**, it receives the error and returns a boolean
89+
90+
::: warning
91+
When defining `condition` as a function, it must be done in a test file directly, not in a configuration file (configurations are serialized for worker threads).
92+
:::
93+
94+
#### RegExp condition (in config file):
95+
96+
```ts
97+
export default defineConfig({
98+
test: {
99+
retry: {
100+
count: 2,
101+
condition: /ECONNREFUSED|ETIMEDOUT/i, // Retry on connection/timeout errors
102+
},
103+
},
104+
})
105+
```
106+
107+
#### Function condition (in test file):
108+
109+
```ts
110+
import { describe, test } from 'vitest'
111+
112+
describe('tests with advanced retry condition', () => {
113+
test('with function condition', { retry: { count: 2, condition: error => error.message.includes('Network') } }, () => {
114+
// test code
115+
})
116+
})
117+
```
118+
119+
## Test File Override
120+
121+
You can also define retry options per test or suite in test files:
122+
123+
```ts
124+
import { describe, test } from 'vitest'
125+
126+
describe('flaky tests', {
127+
retry: {
128+
count: 2,
129+
delay: 100,
130+
},
131+
}, () => {
132+
test('network request', () => {
133+
// test code
134+
})
135+
})
136+
137+
test('another test', {
138+
retry: {
139+
count: 3,
140+
condition: error => error.message.includes('timeout'),
141+
},
142+
}, () => {
143+
// test code
144+
})
145+
```

‎packages/runner/src/run.ts‎

Lines changed: 75 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import type { Awaitable } from '@vitest/utils'
1+
import type { Awaitable, TestError } from '@vitest/utils'
22
import type { DiffOptions } from '@vitest/utils/diff'
33
import type { FileSpecification, VitestRunner } from './types/runner'
44
import type {
@@ -34,6 +34,42 @@ const now = globalThis.performance ? globalThis.performance.now.bind(globalThis.
3434
const unixNow = Date.now
3535
const { clearTimeout, setTimeout } = getSafeTimers()
3636

37+
/**
38+
* Normalizes retry configuration to extract individual values.
39+
* Handles both number and object forms.
40+
*/
41+
function getRetryCount(retry: number | { count?: number } | undefined): number {
42+
if (retry === undefined) {
43+
return 0
44+
}
45+
if (typeof retry === 'number') {
46+
return retry
47+
}
48+
return retry.count ?? 0
49+
}
50+
51+
function getRetryDelay(retry: number | { delay?: number } | undefined): number {
52+
if (retry === undefined) {
53+
return 0
54+
}
55+
if (typeof retry === 'number') {
56+
return 0
57+
}
58+
return retry.delay ?? 0
59+
}
60+
61+
function getRetryCondition(
62+
retry: number | { condition?: RegExp | ((error: TestError) => boolean) } | undefined,
63+
): RegExp | ((error: TestError) => boolean) | undefined {
64+
if (retry === undefined) {
65+
return undefined
66+
}
67+
if (typeof retry === 'number') {
68+
return undefined
69+
}
70+
return retry.condition
71+
}
72+
3773
function updateSuiteHookState(
3874
task: Task,
3975
name: keyof SuiteHooks,
@@ -266,6 +302,32 @@ async function callCleanupHooks(runner: VitestRunner, cleanups: unknown[]) {
266302
}
267303
}
268304

305+
/**
306+
* Determines if a test should be retried based on its retryCondition configuration
307+
*/
308+
function passesRetryCondition(test: Test, errors: TestError[] | undefined): boolean {
309+
const condition = getRetryCondition(test.retry)
310+
311+
if (!errors || errors.length === 0) {
312+
return false
313+
}
314+
315+
if (!condition) {
316+
return true
317+
}
318+
319+
const error = errors[errors.length - 1]
320+
321+
if (condition instanceof RegExp) {
322+
return condition.test(error.message || '')
323+
}
324+
else if (typeof condition === 'function') {
325+
return condition(error)
326+
}
327+
328+
return false
329+
}
330+
269331
export async function runTest(test: Test, runner: VitestRunner): Promise<void> {
270332
await runner.onBeforeRunTask?.(test)
271333

@@ -300,7 +362,7 @@ export async function runTest(test: Test, runner: VitestRunner): Promise<void> {
300362

301363
const repeats = test.repeats ?? 0
302364
for (let repeatCount = 0; repeatCount <= repeats; repeatCount++) {
303-
const retry = test.retry ?? 0
365+
const retry = getRetryCount(test.retry)
304366
for (let retryCount = 0; retryCount <= retry; retryCount++) {
305367
let beforeEachCleanups: unknown[] = []
306368
try {
@@ -412,9 +474,19 @@ export async function runTest(test: Test, runner: VitestRunner): Promise<void> {
412474
}
413475

414476
if (retryCount < retry) {
415-
// reset state when retry test
477+
const shouldRetry = passesRetryCondition(test, test.result.errors)
478+
479+
if (!shouldRetry) {
480+
break
481+
}
482+
416483
test.result.state = 'run'
417484
test.result.retryCount = (test.result.retryCount ?? 0) + 1
485+
486+
const delay = getRetryDelay(test.retry)
487+
if (delay > 0) {
488+
await new Promise(resolve => setTimeout(resolve, delay))
489+
}
418490
}
419491

420492
// update retry info

‎packages/runner/src/types.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,12 @@ export type {
2020
InferFixturesTypes,
2121
OnTestFailedHandler,
2222
OnTestFinishedHandler,
23+
Retry,
2324
RunMode,
2425
RuntimeContext,
2526
SequenceHooks,
2627
SequenceSetupFiles,
28+
SerializableRetry,
2729
Suite,
2830
SuiteAPI,
2931
SuiteCollector,

‎packages/runner/src/types/runner.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type {
44
ImportDuration,
55
SequenceHooks,
66
SequenceSetupFiles,
7+
SerializableRetry,
78
Suite,
89
TaskEventPack,
910
TaskResultPack,
@@ -36,7 +37,7 @@ export interface VitestRunnerConfig {
3637
maxConcurrency: number
3738
testTimeout: number
3839
hookTimeout: number
39-
retry: number
40+
retry: SerializableRetry
4041
includeTaskLocation?: boolean
4142
diffOptions?: DiffOptions
4243
}

‎packages/runner/src/types/tasks.ts‎

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -87,10 +87,12 @@ export interface TaskBase {
8787
*/
8888
result?: TaskResult
8989
/**
90-
* The amount of times the task should be retried if it fails.
90+
* Retry configuration for the task.
91+
* - If a number, specifies how many times to retry
92+
* - If an object, allows fine-grained retry control
9193
* @default 0
9294
*/
93-
retry?: number
95+
retry?: Retry
9496
/**
9597
* The amount of times the task should be repeated after the successful run.
9698
* If the task fails, it will not be retried unless `retry` is specified.
@@ -461,18 +463,70 @@ type ChainableTestAPI<ExtraContext = object> = ChainableFunction<
461463

462464
type TestCollectorOptions = Omit<TestOptions, 'shuffle'>
463465

466+
/**
467+
* Retry configuration for tests.
468+
* Can be a number for simple retry count, or an object for advanced retry control.
469+
*/
470+
export type Retry = number | {
471+
/**
472+
* The number of times to retry the test if it fails.
473+
* @default 0
474+
*/
475+
count?: number
476+
/**
477+
* Delay in milliseconds between retry attempts.
478+
* @default 0
479+
*/
480+
delay?: number
481+
/**
482+
* Condition to determine if a test should be retried based on the error.
483+
* - If a RegExp, it is tested against the error message
484+
* - If a function, called with the TestError object; return true to retry
485+
*
486+
* NOTE: Functions can only be used in test files, not in vitest.config.ts,
487+
* because the configuration is serialized when passed to worker threads.
488+
*
489+
* @default undefined (retry on all errors)
490+
*/
491+
condition?: RegExp | ((error: TestError) => boolean)
492+
}
493+
494+
/**
495+
* Serializable retry configuration (used in config files).
496+
* Functions cannot be serialized, so only string conditions are allowed.
497+
*/
498+
export type SerializableRetry = number | {
499+
/**
500+
* The number of times to retry the test if it fails.
501+
* @default 0
502+
*/
503+
count?: number
504+
/**
505+
* Delay in milliseconds between retry attempts.
506+
* @default 0
507+
*/
508+
delay?: number
509+
/**
510+
* Condition to determine if a test should be retried based on the error.
511+
* Must be a RegExp tested against the error message.
512+
*
513+
* @default undefined (retry on all errors)
514+
*/
515+
condition?: RegExp
516+
}
517+
464518
export interface TestOptions {
465519
/**
466520
* Test timeout.
467521
*/
468522
timeout?: number
469523
/**
470-
* Times to retry the test if fails. Useful for making flaky tests more stable.
471-
* When retries is up, the last test error will be thrown.
472-
*
524+
* Retry configuration for the test.
525+
* - If a number, specifies how many times to retry
526+
* - If an object, allows fine-grained retry control
473527
* @default 0
474528
*/
475-
retry?: number
529+
retry?: Retry
476530
/**
477531
* How many times the test will run again.
478532
* Only inner tests will repeat if set on `describe()`, nested `describe()` will inherit parent's repeat by default.

0 commit comments

Comments
 (0)