Skip to content

WPT reporting (initial PR)#4257

Merged
npaun merged 2 commits intomainfrom
npaun/wpt-reporting
Jun 4, 2025
Merged

WPT reporting (initial PR)#4257
npaun merged 2 commits intomainfrom
npaun/wpt-reporting

Conversation

@npaun
Copy link
Member

@npaun npaun commented Jun 4, 2025

This is the initial PR to enable WPT reporting, so we can add our results to wpt.fyi.

Here's what we have so far:

  • If GEN_TEST_REPORT=1 is set when running a WPT test, we get a wpt report compatible JSON result.
  • If GEN_TEST_STATS=1 is set when running a WPT test, we get a Markdown table row that summarizes the results.
  • We already have GEN_TEST_CONFIG=1 that creates a blank template for test configuration, listing each test in a WPT module.

What's missing - I plan to add these in further PRs later

  • Handle variants: Each variant declared in a // META comment at the top of a file should appear as a separate test. The test URL then gets ?variant=whatev aqdded as a query param
  • Minor detail: set isExpectedFailure correctly in the report. I hardcoded it to true, which is correct in main but might not be in local dev.
  • A script to run all these WPT tests under Bazel and gather the output into one report.
    • GEN_TEST_REPORT: Fill out some metadata about workerd version/commit and add all results. Upload to wpt.fyi
    • GEN_TEST_STATS: Append all of these table rows into one table and put it in the action output somewhere so we can get a human readable summary

@npaun npaun requested review from a team, anonrig and jasnell June 4, 2025 22:14
@npaun npaun requested review from a team as code owners June 4, 2025 22:14
Copy link
Contributor

@windsurf-bot windsurf-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 To request another review, post a new comment with "/windsurf-review".

@npaun npaun merged commit 7ab68a4 into main Jun 4, 2025
18 checks passed
@npaun npaun deleted the npaun/wpt-reporting branch June 4, 2025 22:35
for (const subtest of this.subtests) {
const err = subtest.error;

if (err instanceof Error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have Error.isError() now :-))

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will take care of it in the next PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants