fix(test runner): failed + skipped = flaky#26385

Merged

dgozman merged 1 commit intomicrosoft:mainfrom

dgozman:tr-fail-skip-flaky

Aug 9, 2023

Contributor

dgozman commented Aug 9, 2023

Fixes #17652.

yury-s approved these changes

View reviewed changes

Contributor

github-actions bot commented Aug 9, 2023

Test results for "tests 1"

7 failed
❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry
❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry
❌ [playwright-test] › expect-poll.spec.ts:222:5 › should show intermediate result for poll that spills over test time
❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry
❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry
❌ [playwright-test] › reporter-html.spec.ts:1452:7 › merged › labels › filter should update stats
❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry

8 flaky
⚠️ [chromium] › library/tracing.spec.ts:239:5 › should not include trace resources from the previous chunks
⚠️ [firefox] › page/page-goto.spec.ts:482:3 › should succeed on url bar navigation when there is pending navigation
⚠️ [playwright-test] › ui-mode-trace.spec.ts:53:5 › should merge web assertion events
⚠️ [playwright-test] › ui-mode-trace.spec.ts:126:5 › should show snapshots for sync assertions
⚠️ [chromium] › library/tracing.spec.ts:239:5 › should not include trace resources from the previous chunks
⚠️ [webkit] › library/browsercontext-reuse.spec.ts:50:1 › should reset serviceworker
⚠️ [playwright-test] › ui-mode-trace.spec.ts:53:5 › should merge web assertion events
⚠️ [playwright-test] › ui-mode-trace.spec.ts:126:5 › should show snapshots for sync assertions

25044 passed, 583 skipped
✔️✔️✔️

Details

❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry

Error: expect(received).toBe(expected) // Object.is equality

Expected: 1
Received: 0

  125 |     `,
  126 |   }, { retries: 1 });
> 127 |   expect(result.exitCode).toBe(1);
      |                           ^
  128 |   expect(result.passed).toBe(2);
  129 |   expect(result.flaky).toBe(1);
  130 |   expect(result.failed).toBe(1);

    at /Users/runner/work/playwright/playwright/tests/playwright-test/test-serial.spec.ts:127:27

❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry

Error: expect(received).toBe(expected) // Object.is equality

Expected: 1
Received: 0

  125 |     `,
  126 |   }, { retries: 1 });
> 127 |   expect(result.exitCode).toBe(1);
      |                           ^
  128 |   expect(result.passed).toBe(2);
  129 |   expect(result.flaky).toBe(1);
  130 |   expect(result.failed).toBe(1);

    at /home/runner/work/playwright/playwright/tests/playwright-test/test-serial.spec.ts:127:27

❌ [playwright-test] › expect-poll.spec.ts:222:5 › should show intermediate result for poll that spills over test time

Error: expect(received).toContain(expected) // indexOf

Expected substring: "Expected: 2"
Received string:    "
Running 1 test using 1 worker·
  ✘  1 a.spec.ts:3:11 › should fail (2.0s)··
  1) a.spec.ts:3:11 › should fail ──────────────────────────────────────────────────────────────────·
    Test timeout of 2000ms exceeded.·
  1 failed
    a.spec.ts:3:11 › should fail ───────────────────────────────────────────────────────────────────
"

  230 |   }, { timeout: 2000 });
  231 |   expect(result.exitCode).toBe(1);
> 232 |   expect(result.output).toContain('Expected: 2');
      |                         ^
  233 |   expect(result.output).toContain('Received: 3');
  234 | });
  235 |

    at /home/runner/work/playwright/playwright/tests/playwright-test/expect-poll.spec.ts:232:25

❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry

Error: expect(received).toBe(expected) // Object.is equality

Expected: 1
Received: 0

  125 |     `,
  126 |   }, { retries: 1 });
> 127 |   expect(result.exitCode).toBe(1);
      |                           ^
  128 |   expect(result.passed).toBe(2);
  129 |   expect(result.flaky).toBe(1);
  130 |   expect(result.failed).toBe(1);

    at /home/runner/work/playwright/playwright/tests/playwright-test/test-serial.spec.ts:127:27

❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry

Error: expect(received).toBe(expected) // Object.is equality

Expected: 1
Received: 0

  125 |     `,
  126 |   }, { retries: 1 });
> 127 |   expect(result.exitCode).toBe(1);
      |                           ^
  128 |   expect(result.passed).toBe(2);
  129 |   expect(result.flaky).toBe(1);
  130 |   expect(result.failed).toBe(1);

    at /home/runner/work/playwright/playwright/tests/playwright-test/test-serial.spec.ts:127:27

❌ [playwright-test] › reporter-html.spec.ts:1452:7 › merged › labels › filter should update stats

Error: expect(received).toBe(expected) // Object.is equality

Expected: "Total time: 1270ms"
Received: "Total time: 1.3s"

  1774 |         await expect(page.locator('.chip', { hasText: 'c.test.js' })).toHaveCount(1);
  1775 |
> 1776 |         await flakyButton.click();
       |                                 ^
  1777 |
  1778 |         await expect(searchInput).toHaveValue('@regression @flaky');
  1779 |         await expect(page).toHaveURL(/@regression%20@flaky/);

    at checkTotalDuration (D:\a\playwright\playwright\tests\playwright-test\reporter-html.spec.ts:1776:33)
    at D:\a\playwright\playwright\tests\playwright-test\reporter-html.spec.ts:1792:9

❌ [playwright-test] › test-serial.spec.ts:98:5 › test.describe.serial should work with retry

Error: expect(received).toBe(expected) // Object.is equality

Expected: 1
Received: 0

  125 |     `,
  126 |   }, { retries: 1 });
> 127 |   expect(result.exitCode).toBe(1);
      |                           ^
  128 |   expect(result.passed).toBe(2);
  129 |   expect(result.flaky).toBe(1);
  130 |   expect(result.failed).toBe(1);

    at D:\a\playwright\playwright\tests\playwright-test\test-serial.spec.ts:127:27

Merge workflow run.


          fix(test runner): failed + skipped = flaky

52ca875

dgozman force-pushed the tr-fail-skip-flaky branch from 89f7d33 to 52ca875 Compare

August 9, 2023 22:04

Contributor

github-actions bot commented Aug 9, 2023

Test results for "tests 1"

13 flaky
⚠️ [chromium] › page/page-event-request.spec.ts:101:3 › should report navigation requests and responses handled by service worker
⚠️ [chromium] › library/tracing.spec.ts:239:5 › should not include trace resources from the previous chunks
⚠️ [chromium] › library/tracing.spec.ts:395:14 › should produce screencast frames fit
⚠️ [chromium] › page/page-event-request.spec.ts:101:3 › should report navigation requests and responses handled by service worker
⚠️ [chromium] › page/page-event-request.spec.ts:130:3 › should report navigation requests and responses handled by service worker with routing
⚠️ [chromium] › library/tracing.spec.ts:239:5 › should not include trace resources from the previous chunks
⚠️ [firefox] › page/page-event-request.spec.ts:162:3 › should return response body when Cross-Origin-Opener-Policy is set
⚠️ [playwright-test] › ui-mode-trace.spec.ts:22:5 › should merge trace events
⚠️ [playwright-test] › ui-mode-trace.spec.ts:53:5 › should merge web assertion events
⚠️ [playwright-test] › ui-mode-trace.spec.ts:126:5 › should show snapshots for sync assertions
⚠️ [chromium] › library/tracing.spec.ts:239:5 › should not include trace resources from the previous chunks
⚠️ [webkit] › library/browsercontext-reuse.spec.ts:50:1 › should reset serviceworker
⚠️ [playwright-test] › ui-mode-trace.spec.ts:80:5 › should merge screenshot assertions

25046 passed, 583 skipped
✔️✔️✔️

Merge workflow run.

dgozman merged commit cadc315 into microsoft:main

Germandrummer92 pushed a commit to OctoMind-dev/playwright that referenced this pull request


          fix(test runner): failed + skipped = flaky (microsoft#26385)

Fixes microsoft#17652.

Manvel mentioned this pull request

[BUG] Test mistakenly identified as flaky #28322

Closed

1 task

This was referenced Apr 24, 2024

fix(reporters): properly determine flaky status for serial mode #30529

Merged

fix(playwright): test unexpected result if failed and skipped #30276

Closed

dgozman added a commit that referenced this pull request


          fix(reporters): properly determine flaky status for serial mode (#30529)

dc06652

There are plenty of edge cases in this area:
- interrupted test run;
- did not run because of serial mode failure;
- failed before `test.skip()` call (e.g. in `beforeEach`) in one of the
retries;
- and more...

Related issues: #28322, #28321, #27455, #17652.
Prior changes: #27762, #26385, #28360, probably more.

There is still some duplication between `outcome()` and similar logic in
`base.ts`, which might be deduped in a follow-up.

Fixes #28322.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet