[BUG] Serial + Retry : Failed test is marked as flaky even if it never passes #27455
Description
System info
- Playwright Version: [v1.38.0]
- Operating System: [All, Windows 11, Ubuntu 20, macOS 13.2, etc.]
- Browser: [All, Chromium, Firefox, WebKit]
Source code
import {expect, test} from '@playwright/test';
test.describe.serial("Tests", () => {
// Flaky test
test("First test", () => {
expect(Math.random()).toBeGreaterThan(0.3); // some flaky step
});
test("Second test", () => {
expect(true).toEqual(false);
});
});Description
The second test is marked as flaky if tests are in describe.serial block with "retries: 1" settled in tests config file.
Steps
Execute the test until:
- The second test is executed at least once and the first test failed on the last retry
Expected
1 failed
[chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (Failed) ──────────
1 passed (648ms)
Actual
If the second test is executed at least once and the first test failed on the last retry:
Running 2 tests using 1 worker
✓ 1 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (21ms)
✘ 2 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (11ms)
✘ 3 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (retry #1) (23ms)
- 4 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (retry #1)
...
2 flaky
[chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test ────────────
[chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test ──────────
If the second test is executed on the last retry:
Running 2 tests using 1 worker
✓ 1 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (21ms)
✘ 2 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (11ms)
✓ 3 [chromium] › test-serial-retry.spec.ts:4:7 › Tests › First test (retry #1) (21ms)
✘ 4 [chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test (retry #1) (11ms)
...
1 failed
[chromium] › test-serial-retry.spec.ts:8:7 › Tests › Second test ──────────
1 passed (648ms)