Not set in stone, feedback welcome.

Follows the previous proposal and the meeting notes.

Prior art: #34089, #38401.

Notable changes

• Use @playwright/test rather than jest + playwright. The playwright test runner offers more features out of the box specifically for e2e testing. It's also one of the primary focuses of the playwright team (seems like they're now branding playwright as a test framework).
• I created a separated package e2e-test-utils-playwright for all the utils used in Playwright. The new tests written in Playwright should be in test/e2e/specs.
• This PR adds a sanity.spec.js just to show that it works as expected. It should be removed once we have other tests migrated.
• I haven't written any doc yet, but I have some ideas of what we could do during the migration (written down below). We can discuss and finalize it before publishing them to the docs folder.
• It is still not clear how to run the playwright tests in Core, but it should be doable.

Migration steps

1. Choose a test suite to migrate in packages/e2e-tests/specs, rename .test.js into .spec.js and put it in the same folder structure inside test/e2e/specs.
2. Require the test helpers from @wordpress/e2e-test-utils-playwright:

   const { test, expect } = require( '@wordpress/e2e-test-utils-playwright' );

3. Change all occurrences of describe, beforeAll, beforeEach, afterEach and afterAll with the test. prefix. For instance, describe is turning into test.describe.
4. Use the fixtures API to require previously global variables like page and browser.
5. Delete all the imports of e2e-test-utils, instead use the fixtures API to get the pageUtils and requestUtils directly. (However, pageUtils is not allowed in beforeAll and afterAll, rewrite them using requestUtils instead.)
6. If there's a missing util, go ahead and migrate it. Copy the existing file in e2e-test-utils and paste it in the page or request folder in e2e-test-utils-playwright depending on whether it's a page util or a request util.
7. Global page and browser are available in PageUtils's this.page and this.browser. You can get autocomplete and type inference by adding @this {import('./').PageUtils} to the JSDoc.
8. All the other utils in the same class are available in this and bound to the same instance. You can remove the internal imports and just use this to access them.
9. Import the newly migrated util in index.ts, and put it inside the PageUtils/RequestUtils class as an instance field.
10. You can now use the util inside the pageUtils or the requestUtils fixture in test/e2e.
11. Manually migrate other details in the tests following the proposed best practices below. Note that even though the differences in the API of playwright and puppeteer are similar, some manual changes are still required.

Proposed best practices

Forbid $, use locator instead

In fact, any API that returns ElementHandle is discouraged. This includes $, $$, $eval, $$eval, etc. Locator is a much better API and can be used with playwright's assertions. This also works great with Page Object Model since that locator is lazy and doesn't return a promise.

Use accessible selectors

This PR introduces a new selector engine role-selector. It enables us to write accessible queries without having to rely on internal implementations. It's an experimental library and could be swapped out if playwright supports it natively instead. The syntax should be straightforward and looks like this:

// Select a button with the accessible name "Hello World".
page.locator( 'role=button[name="Hello World"]' );

It can also be chained with built-in selector engines to perform complex queries:

// Select a button with a name ends with `Back` and is visible on the screen.
page.locator( 'role=button[name=/Back$/] >> visible=true' );
// Select a button with name "View options" under `#some-section`.
page.locator( 'css=#some-section >> role=button[name="View options"]' );

role-selector under the hood uses @testing-library/dom to compute the query the elements and compute the accessible attributes.

Selectors are strict by default

To encourage better practices for querying elements, selectors are strict by default, meaning that it will throw an error if the query returns more than one element.

Don't overload test-utils, inline simple utils

e2e-test-utils are too bloated with too many utils. Most of them are simple enough to be inlined directly in tests. With the help of accessible selectors, simple utils are easier to write now. For utils that only take place on a certain page, use Page Object Model instead (with an exception of clearing states with rest which are better placed in e2e-test-utils). Otherwise, only create an util if the action is complex and repetitive enough.

Favor Page Object Model over utils

As mentioned above, Page Object Model is the preferred way to create reusable utility functions on a certain page. (You can see an inline POM example in #38570 (comment))

The rationale behind using a POM is to group utils under namespaces to be easier to discover and use. In fact, PageUtils in e2e-test-utils` package is also a POM, which avoids the need for global variables, and utils can reference each other easily.

Restify actions to clear or set states.

It's slow to set states manually before or after tests, especially when they're repeated multiple times between tests. It's recommended to set them via API calls. Use requestUtils.rest and requestUtils.batchRest instead to call the REST API. We should still add a test for manually setting them, but that should only be tested once.

Avoid global variables

page and browser are not exposed as global variables, this makes it easier to work with when we have multiple pages/tabs in the same test, or if we want to run multiple tests in parallel. @playwright/test has the concept of fixtures which allows us to inject page, browser, and other parameters into the tests.

By default, @playwright/test will use different page instances for each test to isolate them. However, the nature of WordPress means most of our tests depend on the server state and we can't easily isolate them. We instead run the tests in serial to prevent race-condition.

Make explicit assertions

We can insert as many assertions in one test as needed. It's better to make explicit assertions whenever possible. For instance, if we want to assert that a button exists before clicking on it, we can do expect( locator ).toBeVisible() before performing locator.click(). This makes the tests flow better and easier to read.

Testing Instructions

# Be sure to install the dependencies first
npm install

# Run all tests
npm run test-e2e:playwright

# Run individual test
npm run test-e2e:playwright -- <path/to/test>

# Run in headed mode
npm run test-e2e:playwright -- --headed

# Run in debug mode
npm run test-e2e:playwright -- --debug

Types of changes

New feature

Checklist:

• My code is tested.
• My code follows the WordPress code style.
• My code follows the accessibility standards.
• I've tested my changes with keyboard and screen readers.
• My code has proper inline documentation.
• I've included developer documentation if appropriate.
• I've updated all React Native files affected by any refactorings/renamings in this PR (please manually search all *.native.js files for terms that need renaming or removal).
• I've updated related schemas if appropriate.