System info

• Playwright Version: v1.39.0, v1.40.0
• Operating System: All
• Browser: n/a
• Other info:

Source code

• I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

https://github.com/currents-dev/pw-issue-2023-11-23-001

Steps

• Checkout the demo repo, follow README instructions
• Run the test for 1.39
• Run the test for 1.40

Expected

The following test suite:

import { expect, test } from "@playwright/test";

test.describe.configure({ mode: "serial", retries: 1 });

test("A", async ({ page }, { retry }) => {
  if (retry === 0) {
    throw new Error("oh!");
  }
  if (retry === 1) {
    expect(true).toBe(true);
  }

});

test("B", async ({ page }, { retry }) => {
  throw new Error("oh!");
});

Is supposed to produce the following outcome:

• Test A results status: failed, passed

• Test B results status: skipped, failed

• onTestBegin should be invoked for every attempt of every test in a serial group as in 1.39

• all tests should exhaust all attempts in a serial group

Actual

1.40 runs serial tests group differently and produces different results:

• Test A results status: failed, passed

• Test B results status: failed

• 1.40 doesn't invoke onTestBegin for test B when A fails

• 1.40 doesn't retry test B and "swallows" the last retry