System info

• Playwright Version: v1.39.0
• Operating System: All
• Browser: All
• Other info:

Source code

• I provided the exact source code that allows reproducing the issue locally.

Test file (self-contained)

import { expect, test } from "@playwright/test";

test.describe.configure({ mode: "serial", retries: 3 });

test("A", async ({ page }, { retry }) => {
  if (retry === 0 || retry === 2) {
    throw new Error("oh!");
  }
  if (retry === 1 || retry === 3) {
    expect(true).toBe(true);
  }
});

test("B", async ({ page }, { retry }) => {
  throw new Error("oh!");
});

Steps

• Run the test npx playwright test
• See both tests identified as flaky
• CLI exit status is 0

1.39 produces the following outcome for attempts:

• Test A: failed, passed, failed, passed
• Test B: skipped, failed, skipped, failed

Expected

• Test B should not be flaky, it should be failed. All the non-skipped attempts do not match the expected status - i.e. the test never produces the expected passed status.
• CLI exit status should not be 0 to mark

Actual

• Both tests are identified as flaky
• CLI exit status is 0 and the CI passes