feat(test-runner): test filter #33049

muhqu · 2024-10-10T18:28:28Z

Adds a new way for playwright users to filter the list of test cases using a function. It's similar to grep, but instead of a RegExp you can use a function.

With this function it ~~should~~ is be possible to implement a custom sharding mechanism as a user of playwright.

New property: TestConfig.filter

type: ?<[TestFilter]|[Array]<[TestFilter]>>

Filter test cases by function. TestFilter can either be predicate function or an object with filterTests or filterTestGroups methods.

Usage

import { defineConfig } from '@playwright/test';

export default defineConfig({
  filter: test => test.title === 'some test',
});

Or...

export default defineConfig({
  filter: {
    filterTests: tests => tests.filter((test, index) => index % 2 === 0)
  },
});

Or...

export default defineConfig({
  filter: {
    filterTestGroups: testgroups => {
      return testgroups.filter((testgroups, index) => index % 2 === 0)
    },
  },
});

packages/playwright/types/test.d.ts

muhqu · 2024-10-11T08:33:29Z

packages/playwright/types/test.d.ts

 import type { APIRequestContext, Browser, BrowserContext, BrowserContextOptions, Page, LaunchOptions, ViewportSize, Geolocation, HTTPCredentials, Locator, APIResponse, PageScreenshotOptions } from 'playwright-core';
 export * from 'playwright-core';

+import type { TestCase } from './testReporter';


I'm not sure whether it is okay to simply cross reference TestCase here…

It is and it isn't, but let's solve other bits before resolving it.

packages/playwright/types/test.d.ts

github-actions · 2024-10-11T15:28:37Z

Test results for "tests 1"

3 flaky

⚠️ [playwright-test] › ui-mode-test-ct.spec.ts:215:5 › should watch component via util @macos-latest-node18-1
⚠️ [installation tests] › playwright-component-testing.spec.ts:55:5 › pnpm: JSX inside a @playwright/test should work @package-installations-macos-latest @playwright/test
⚠️ [playwright-test] › ui-mode-test-setup.spec.ts:22:5 › should run global setup and teardown @ubuntu-latest-node22-1

35917 passed, 620 skipped
✔️✔️✔️

Merge workflow run.

pavelfeldman · 2024-10-12T01:59:24Z

When is testgroups.filter useful?

muhqu · 2024-10-12T15:42:44Z

Test group filtering is useful to make sure tests, which playwright wants to executed together, stay together (e.g. when they need to run sequential…). The logic which tests should be grouped together stays internal to playwright and all a user of playwright should know is which tests are grouped together, not why.

If we do not expose test groups, users of playwright need to make assumptions of which tests should belong together, which is suboptimal as it might break playwrights contracts (e.g. when tests need to run in sequence…)

dgozman · 2024-10-15T08:01:53Z

@muhqu Thank you for the explanation. We've discussed this proposal, and here are some thoughts:

It seems like operating on test groups is the only viable solution among the ones outlined above.
We would really like a low level API to also be able to control the order of test execution. For example, one could imagine a strategy that would "run last failures first, and only if all of them pass - run all other tests". Ideally, we want strategies like this to be implementable on top of the proposed API.
All of this is complicated by dependencies, serial mode, worker hashes, and probably more things that I didn't think about yet.

Given the above, I'd recommend to file an issue first, where we can outline the requirements and considerations before arriving to an API. We judge the interest of the community in the topic by engagement numbers on the issue, so that would be important for us.

Additionally, given an issue, we will also be able to experiment with some ideas from our side, instead of blocking you for a long time or asking to try things that would eventually lead to nowhere. For example, one of the ideas we discussed is allowing tests with different worker hashes in the same worker, which might simplify the custom scheduling API quite a bit.

muhqu · 2024-10-15T17:57:34Z

@dgozman ok, this helps me understand what was meant by “low level api”. But given your list of requirements is clearly larger than mine, I think it makes more sense you come up with a feature request to make the test runner architecture more extensible… e.g. pluggable test runner strategies… that’s beyond what I was trying to solve.

lukpsaxo · 2024-11-29T08:38:12Z

@dgozman I'd love to see this not die - there is an issue linked here - #17969 with 41 up votes. It would be great to see the team engage in the issue and..

outline the requirements and considerations before arriving to an API

we bet on muhqu's PR being merged, but we are at the point where there is a 20 minute difference between agents of our test runs and we would like to know if we should go back to our approach of splitting the tests ourselves and writing out a different config file per shard.

feat(test-runner): test filter

58242e3

This was referenced Oct 10, 2024

test(test-runner): test round-robin sharding with stable-test-runner #33032

Closed

feat(test runner): improve sharding algorithm to better spread similar tests among shards #30962

Closed

muhqu changed the title ~~feat(test-runner): test filter~~ WIP feat(test-runner): test filter Oct 10, 2024

muhqu commented Oct 10, 2024

View reviewed changes

packages/playwright/types/test.d.ts Outdated Show resolved Hide resolved