Skip to content

[wip] testing flakiness of firefox/chrome tests in GHA#4936

Closed
dbjorge wants to merge 16 commits intodevelopfrom
gha-test-firefox-flakiness-testing
Closed

[wip] testing flakiness of firefox/chrome tests in GHA#4936
dbjorge wants to merge 16 commits intodevelopfrom
gha-test-firefox-flakiness-testing

Conversation

@dbjorge
Copy link
Copy Markdown
Contributor

@dbjorge dbjorge commented Nov 12, 2025

not intended to merge, this is just #4919 with those 2 jobs running 10x each

@dbjorge dbjorge closed this Nov 12, 2025
dbjorge added a commit that referenced this pull request Nov 12, 2025
#4938)

In #4919, we found that the `test_firefox` job was unacceptably flaky in
the `preload-cssom` tests. I believe this is because the GHA runners are
slightly more resource-constrained than the circleci ones, which broke
an assumption in `axe.testUtils.addStyleSheet` that adding a `<style>`
element would result in `document.styleSheets` being updated accordingly
within 100ms (in Chrome this happens instantly, but in Firefox there is
a delay). I updated the test util to poll until it actually *sees*
`document.styleSheets` update as expected, and to reject with a more
actionable error if a timeout occurs (rather than letting a test run in
an unexpected state). This also has a minor side-benefit of making the
tests marginally faster on Chrome, since they no longer wait
100ms-per-injected-sheet unnecessarily.

You can see the results of this in
#4936 - the preload-cssom
test_firefox case [only passed 6/10 runs without this
fix](https://github.com/dequelabs/axe-core/pull/4936/checks?sha=1516bacef0b7e039030adbd4d55c1a627f9899fb)
and [passed 10/10 runs with this
fix](https://github.com/dequelabs/axe-core/pull/4936/checks?sha=339c2580bb82f2f7833b8e2447bfeff92354350e).

The updates in `preload-cssom.js` turned out to be unnecessary to fix
the issue, but they make errors in the test more actionable and make the
test more consistent with others, so I left them in.

No QA required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants