ci(nightly): port testing to github actions#4918
Conversation
bd73665 to
9bec47e
Compare
9bec47e to
48d7f26
Compare
| }); | ||
|
|
||
| before(done => { | ||
| driver = getWebdriver(); |
There was a problem hiding this comment.
At one point I thought I had a race condition on things. Ended up looking at this and realized we had duplicate before blocks that aren't necessary. So got them together for better clarity in the processing order.
There was a problem hiding this comment.
Pull Request Overview
This PR migrates nightly browser testing from CircleCI to GitHub Actions while improving browser binary and driver configuration flexibility through environment variables.
- Removes CircleCI nightly test jobs and workflow configuration
- Adds a new GitHub Actions workflow for nightly tests with scheduled and manual triggers
- Updates test files to support configurable browser binaries and ChromeDriver paths via environment variables
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/nightly-tests.yml |
Adds new nightly test workflow with jobs for browsers, ACT rules, and ARIA practices |
.github/actions/install-deps/action.yml |
Creates reusable action for installing dependencies with optional nightly browser versions |
test/integration/full/test-webdriver.js |
Adds environment variable support for Chrome/Firefox binaries and ChromeDriver, fixes typo and headless flag |
test/get-webdriver.js |
Adds environment variable support for Chrome binary and ChromeDriver path |
test/act-rules/act-runner.js |
Consolidates before hooks for cleaner test setup |
.circleci/config.yml |
Removes nightly test jobs and workflow from CircleCI configuration |
Comments suppressed due to low confidence (1)
.github/workflows/nightly-tests.yml:1
- The comment at line 58 in
.github/actions/install-deps/action.ymlindicates 'nightly' mode installs 'beta' for Chrome, but this is misleading. Chrome beta is not the nightly version - Chrome has separate Canary/Dev channels for nightly builds. Consider adding a comment to clarify that 'beta' is the closest available pre-release channel supported by the browser-actions/setup-chrome action.
name: Nightly Tests
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| jobs: | ||
| browsers: | ||
| runs-on: ubuntu-24.04 |
There was a problem hiding this comment.
I'm going to start locking our ubuntu version as well. Just since some packages we may need to install might change names between versions, especially with the unix timestamp things going on.
Extra layer of pinning where feasible to help make sure changes under us don't break us later.
| - name: Run Firefox Nightly Browser Tests | ||
| env: | ||
| FIREFOX_NIGHTLY_BIN: ${{ steps.install-deps.outputs.firefox-path }} | ||
| run: npm run test -- --browsers FirefoxNightly | ||
| - name: Run Chrome Beta Browser Tests |
There was a problem hiding this comment.
So before we would run chrome and firefox in kinda parallel with the command run: npm run test -- --browsers Chrome,FirefoxNightly letting Karma handle how to run it, but this makes it run serially. Any reason why you separated it out like that?
There was a problem hiding this comment.
Eventual reporting to Slack when failures happen. To have it more concisely tell us which browser failed. Since we can check the step that failed when the time comes. At least for right now, it is more clear if things do fail from a nightly which it is from the log. Instead of parsing the log contents to figure out the browser in question.
| }); | ||
|
|
||
| before(done => { | ||
| driver = getWebdriver(); |
There was a problem hiding this comment.
I think this is incorrect. getWebdriver returns a promise as the function returns the new Builder().build() but doesn't await it. I think the prior before being async meant it would return the promise and wait for it to resolve before completing. Now I believe driver is the promise and not the resolved promise value.
There was a problem hiding this comment.
That is not what the prior behavior did. It would put the before function on the microtask queue. Then it would spin up, run the thing and put the promise to the variable. It would not await it, since there was no await in the function body itself.
If we need to wait on this for safety, then there are two ways to handle it. I'll look first into making sure it is done here where both those are the one function since we have the done callback.
There was a problem hiding this comment.
Hu weird. Testing this out it looks like webdriver handles oddities like this. It returns a thenable but doesn't resolve the driver build until much later. When driver.get is called it'll resolve it right there, but if I comment out the driver.get it will resolve much later. So it seems webdriver is ok handling a driver that is still waiting for the promise when we call get later. I didn't know that's how it worked.
There was a problem hiding this comment.
I just double checked the type because I remember looking into this and doing this for a reason... and yes, this build method is NOT asynchronous. Ref: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/selenium-webdriver/index.d.ts#L534-L547
It returns a thenable object so the result can be used as such. But the main call itself is synchronous. So this is doing the exact same behavior as before. The extra async hook was not providing any value.
# Background In order to continue publishing to npm, the org is moving to OIDC tokens. This is setup through GitHub Actions. Which has prompted a conversion from CircleCI to GHA for this repository. In this the two previous PRs, #4918 and #4919, have slowly ported chunks of the dependency workflows over. This PR now provides the deployment support to ship changes from GHA. # Key Changes ## Addition of Deployment Workflow **Concurrency**: One queue per branch. Does not cancel previous runs. The deployment workflow is triggered only on `develop` and `master` commits once the tests have completed. First, the status of the tests is checked before proceeding with any work. If any failures happened, this workflow does not do anything. First, the "Tests" workflow for the associated commit is waited on. About 12 minutes (overhead for network slowness) is allocated, 14 total runtime minutes on the job before it is force killed. If the workflow was successful, the deployment for the branch in question begins. If any other conclusion is found, or none at the end of the timer, the job exits with an error so no deployment occurs. If the branch is `develop`, execution runs autonomously to build and test the package before publishing. Once all is successful, a publish of the next tag is conducted. If the branch is `master`, execution starts with the `prod-hold` job. This is a job that accesses an environment that is defined with "Required Reviewers". Meaning it will not allow the job to continue until someone on that reviewer list has approved it to happen. Once permission is granted, the `prod-deploy` job kicks off. Conducting the same actions as the next release does, but no special tags on the version which makes it `latest` and stable. Once the publish is out, some key information is retrieved as output for the job. Once a successful `prod-deploy` has happened, two jobs kick off. One creates the GitHub Release, which uses the same script as was used in CircleCI. The other sets up NodeJS, waits for the package to be visible on npm if it isn't already, then installs the package globally. After global installation finishes, a few tests of the package that was deployed are run to ensure it is functional. A key part of both workflows, is ensuring the package contents look correct before publishing. To support more expansive testing of the required behavior, a new node script is introduced. Which helps separate the pre-publish and post-publish testing. As well as expands the capability of that test to be more comprehensive. ### New Environment A new environment is introduced, `production-deploy`. This is restricted to only being accessible from the `master` branch. It is configured with required reviewers. So *before* the environment can be accessed in a workflow run, someone from the required reviewer group must manually approve the access in the GitHub Actions UX. This secondary environment is needed since we auto-deploy `next` tags off the `develop` branch. But for `master` with stable deploys, we want to have manual approval before it goes out. Since we can't add the required reviewers to the main environment as that would impact `develop`, this is made to target just stable releases. ## New pre-publish validation script CircleCI is configured with a shell script that did some pre and post deploy validation of the contents. While it is effective for the basics, it was fairly minimal in what it did as a whole. This now has a far more comprehensive suite of checks that runs for this step. The new script pulls key data from the package to build a report. It checks that _all_ contents of the `files` definition exist on the filesystem. It then checks to ensure that the package can be imported in CommonJS. After that, ensuring it is importable (along with all importable files shipped) are capable of that in ESM. Finally, it validates the SRI hashes defined in the sri history are what are computed from the current version. The SRI check only occurs on `master` and `release-` branches. As on `develop` the hash is not updated with every change. This script works by running as a Node ESM script. It pulls key information, does the filesystem check, then links the package to itself. This makes resolving the package pull the linked version on the filesystem instead of the version installed by `@axe-core/webdriverjs` for integration testing. The output of this is logged to the console and compiled into a more easily viewable step summary. The successful output of which can be seen in [a comment on this PR](#4929 (comment)). ## Addition of final tests before deployment of stable One test not added previously, `test_rule_help_version`, which checks for the docs to be live; is now added. As this only impacted stable releases going out, it made sense to reduce the initial scope and introduce this here. The next test added was an explicit SRI validation. This runs on all commits to `master` and `release-*` branches as well as all PRs targeting them. It uses the old validation pathway for quickly getting the check back in place. We can look into what to do with the SRI stuff itself later. ## Removal of CircleCI Publishing Configuration Within this patch, the CircleCI configuration to deploy to NPM is removed since we now want to run only from OIDC on GitHub Actions. The full test suite is not removed with this. It will be removed shortly after this lands when we convert the required checks over to GHA for merging. ## Visual Overviews ### Deploy Workflow <figure><legend>Workflow Control Color Coding</legend> | Color | Element | |-------|---------| | Light Blue - `#e1f5ff` | Start (Trigger) | | Light Red - `#ffe1e1` | End (No Deploy) | | Light Green - `#e1ffe1` | End (Success states) | | Light Yellow - `#fff4e1` | Decision gates & Manual approval | </figure> <figure><legend>Step Color Coding</legend> | Color | Step Name | |-------|-----------| | Blue - `#b3d9ff` | Checkout Code | | Purple - `#d9b3ff` | Install Dependencies | | Pink - `#ffb3d9` | Build Package | | Green - `#b3ffb3` | Validate Package | | Orange - `#ffd9b3` | Publish to NPM | </figure> <details><summary>Click to open diagram</summary> ```mermaid flowchart TD Start([Push to master/develop]) --> WaitTests[wait-for-tests Job] WaitTests --> WT1[Checkout Code] WT1 --> WT2[Wait for Tests Workflow] WT2 --> TestSuccess{Tests Successful?} TestSuccess -->|No| End([End - Tests Failed]) TestSuccess -->|Yes| BranchCheck{Which Branch?} BranchCheck -->|develop| DeployNext[deploy-next Job] BranchCheck -->|master| ProdHold[prod-hold Job] DeployNext --> DN1[Checkout Code] DN1 --> DN2[Install Dependencies] DN2 --> DN3[Build Package<br/>npm prepare & build] DN3 --> DN4[Determine Prerelease Version] DN4 --> DN5[Bump Version] DN5 --> DN6[Validate Package] DN6 --> DN7[Publish to NPM<br/>--tag=next] DN7 --> ValidateNextDeploy[validate-next-deploy Job] ValidateNextDeploy --> VND1[Checkout Code] VND1 --> VND2[Setup Node.js] VND2 --> VND3[Wait for Package on NPM] VND3 --> VND4[Validate Installation of next] VND4 --> EndNext([End - Next Version Validated]) ProdHold --> PH1[Manual Approval Gate] PH1 --> ProdDeploy[prod-deploy Job] ProdDeploy --> PD1[Checkout Code] PD1 --> PD2[Install Dependencies] PD2 --> PD3[Build Package<br/>npm prepare & build] PD3 --> PD4[Validate Package] PD4 --> PD5[Publish to NPM<br/>stable version] PD5 --> PD6[Get Package Data<br/>version & name] PD6 --> CreateRelease[create-github-release Job] PD6 --> ValidateDeploy[validate-deploy Job] CreateRelease --> CR1[Checkout Code] CR1 --> CR2[Install Release Helper<br/>github-release tool] CR2 --> CR3[Download Release Script] CR3 --> CR4[Make Script Executable] CR4 --> CR5[Create GitHub Release] CR5 --> EndRelease([GitHub Release Created]) ValidateDeploy --> VD1[Checkout Code] VD1 --> VD2[Setup Node.js] VD2 --> VD3[Wait for Package on NPM] VD3 --> VD4[Validate Installation of Stable] VD4 --> EndValidate([End - Deploy Validated]) %% Consistent step colors with accessible text style DN1 fill:#b3d9ff,color:#001a33,stroke:#0066cc style PD1 fill:#b3d9ff,color:#001a33,stroke:#0066cc style CR1 fill:#b3d9ff,color:#001a33,stroke:#0066cc style WT1 fill:#b3d9ff,color:#001a33,stroke:#0066cc style VND1 fill:#b3d9ff,color:#001a33,stroke:#0066cc style VD1 fill:#b3d9ff,color:#001a33,stroke:#0066cc style DN2 fill:#d9b3ff,color:#1a0033,stroke:#6600cc style PD2 fill:#d9b3ff,color:#1a0033,stroke:#6600cc style DN3 fill:#ffb3d9,color:#330011,stroke:#cc0066 style PD3 fill:#ffb3d9,color:#330011,stroke:#cc0066 style DN6 fill:#b3ffb3,color:#2200,stroke:#00aa00 style PD4 fill:#b3ffb3,color:#2200,stroke:#00aa00 style DN7 fill:#ffd9b3,color:#331100,stroke:#cc6600 style PD5 fill:#ffd9b3,color:#331100,stroke:#cc6600 %% Decision/Gate styling style Start fill:#e1f5ff,color:#001a33 style End fill:#ffe1e1,color:#330000 style EndNext fill:#e1ffe1,color:#2200 style EndRelease fill:#e1ffe1,color:#2200 style EndValidate fill:#e1ffe1,color:#2200 style PH1 fill:#fff4e1,color:#331100 style TestSuccess fill:#fff4e1,color:#331100 style BranchCheck fill:#fff4e1,color:#331100 ``` </details> ### Validation Script <figure><legend>Color Coding of Scopes</legend> | Color | Scope | |-------|-------| | Dark Gray - `#1a1a1a` | Setup/Teardown | | Blue - `#004d99` | File Existence Check | | Orange/Brown - `#664d00` | CommonJS Compatibility Check | | Purple - `#4d004d` | Importable Check | | Teal - `#004d4d` | SRI Hash Validation | | Gray - `#5c5c5c` | Skipped operations | </figure> <figure><legend>Color Coding of States</legend> | Color | State | |-------|-------| | Green - `#2d5016` | Success | | Red - `#7d1007` | Error/Failure | | Gray - `#5c5c5c` | Skipped | </figure> <details><summary>Click to open diagram</summary> ```mermaid flowchart TD Start([Start Script]) --> Init[Initialize Variables] Init --> FileCheck[File Existence Check] FileCheck --> FileLoop[For each file in pkg.files array] FileLoop --> FileExists{File exists?} FileExists -->|yes| FilePass[✓ Mark as Found] FileExists -->|no| FileFail[✗ Mark as Missing<br/>exitCode++] FilePass --> LinkSetup FileFail --> LinkSetup LinkSetup[Create npm link] -->|success| CJS[CommonJS Compatibility Check] LinkSetup -->|fail| LinkError[Log error and skip remaining checks] CJS --> CJSRequire{Require package successful?} CJSRequire -->|yes| CJSValidate{Validate export type and version exists?} CJSRequire -->|no| CJSFail[✗ CommonJS Failed<br/>exitCode++] CJSValidate -->|yes| CJSPass[✓ CommonJS Compatible] CJSValidate -->|no| CJSFail CJSPass --> Import CJSFail --> Import Import[Importable Check] --> ImportMain{Import main package?} ImportMain -->|success| ValidateMainVersion{Version property exists?} ImportMain -->|fail| ImportMainFail[✗ Not Importable<br/>anyCaught = true] ValidateMainVersion -->|yes| ImportMainPass[✓ Mark as Importable] ValidateMainVersion -->|no| ImportMainFail ImportMainPass --> ImportLoop ImportMainFail --> ImportLoop ImportLoop[For each file in pkg.files] --> FileType{File type?} FileType -->|skip .txt, .d.ts, folders| CheckAnyCaught FileType -->|check resolve| NodeModCheck{Resolves to node_modules?} NodeModCheck -->|yes| ImportNodeFail[✗ Resolves to node_modules<br/>exitCode++] NodeModCheck -->|no| ImportTry{Import successful?} ImportTry -->|yes| CheckVersion{Has version property?} ImportTry -->|no| ImportFileFail[✗ Not Importable<br/>anyCaught = true] CheckVersion -->|yes| ImportPass[✓ Importable] CheckVersion -->|no| ImportFileFail ImportPass --> CheckAnyCaught ImportFileFail --> CheckAnyCaught ImportNodeFail --> CheckAnyCaught CheckAnyCaught{anyCaught true?} -->|yes| ImportExit[exitCode++] CheckAnyCaught -->|no| SRICheck ImportExit --> SRICheck SRICheck[SRI Hash Validation] --> BranchCheck{Branch is master or release-*?} BranchCheck -->|no| SkipSRI[Skip SRI validation] BranchCheck -->|yes| SRILoad[Load sri-history.json<br/>Calculate hashes for axe.js and axe.min.js] SRILoad --> SRICompare{Hash matches expected?} SRICompare -->|yes| SRIPass[✓ Valid SRI] SRICompare -->|no| SRIFail[✗ Invalid SRI<br/>exitCode++] SRIPass --> Cleanup SRIFail --> Cleanup SkipSRI --> Cleanup Cleanup[Unlink npm package] --> End([Exit with exitCode]) LinkError --> CleanupError[Attempt unlink if needed] CleanupError --> End style Start fill:#1a1a1a,stroke:#000,stroke-width:2px,color:#fff style Init fill:#1a1a1a,stroke:#000,stroke-width:2px,color:#fff style End fill:#1a1a1a,stroke:#000,stroke-width:2px,color:#fff style FileCheck fill:#004d99,stroke:#000,stroke-width:2px,color:#fff style FileLoop fill:#004d99,stroke:#000,stroke-width:2px,color:#fff style FileExists fill:#004d99,stroke:#000,stroke-width:2px,color:#fff style FilePass fill:#2d5016,stroke:#000,stroke-width:2px,color:#fff style FileFail fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style LinkSetup fill:#1a1a1a,stroke:#000,stroke-width:2px,color:#fff style LinkError fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style CJS fill:#664d00,stroke:#000,stroke-width:2px,color:#fff style CJSRequire fill:#664d00,stroke:#000,stroke-width:2px,color:#fff style CJSValidate fill:#664d00,stroke:#000,stroke-width:2px,color:#fff style CJSPass fill:#2d5016,stroke:#000,stroke-width:2px,color:#fff style CJSFail fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style Import fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style ImportMain fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style ValidateMainVersion fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style ImportMainPass fill:#2d5016,stroke:#000,stroke-width:2px,color:#fff style ImportLoop fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style FileType fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style NodeModCheck fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style ImportTry fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style CheckVersion fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style CheckAnyCaught fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style ImportExit fill:#4d004d,stroke:#000,stroke-width:2px,color:#fff style ImportPass fill:#2d5016,stroke:#000,stroke-width:2px,color:#fff style ImportMainFail fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style ImportFileFail fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style ImportNodeFail fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style SRICheck fill:#004d4d,stroke:#000,stroke-width:2px,color:#fff style BranchCheck fill:#004d4d,stroke:#000,stroke-width:2px,color:#fff style SRILoad fill:#004d4d,stroke:#000,stroke-width:2px,color:#fff style SRICompare fill:#004d4d,stroke:#000,stroke-width:2px,color:#fff style SRIPass fill:#2d5016,stroke:#000,stroke-width:2px,color:#fff style SRIFail fill:#7d1007,stroke:#000,stroke-width:2px,color:#fff style SkipSRI fill:#5c5c5c,stroke:#000,stroke-width:2px,color:#fff style Cleanup fill:#1a1a1a,stroke:#000,stroke-width:2px,color:#fff style CleanupError fill:#1a1a1a,stroke:#000,stroke-width:2px,color:#fff ``` </details> Fixes: #4912
This patch begins moving the workflows from CircleCI to Github. In order to minimize the reviews, the first chunk is only converting the nightly tests.
Key Changes
Removal of Circle CI Config for Nightly Tests
Since this patch adds support for running nightly tests in GitHub Actions, we no longer need it in Circle CI. It would be wasteful of resources to do the same work in multiple places.
Install Deps Composite Action
This file reduces duplication of steps between multiple jobs and workflows. It is called after a repository is checked out in CI in order to install core OS dependencies, browsers, package dependencies, and starts (when needed) a virtual X server for running headed browsers against for testing.
Nightly Tests
The nightly workflow runs the current beta releases for Firefox and Chrome, WCAG ACT's current version, and the latest aria practices tests.
Webdriver Configuration
Previously,
browser-driver-managerwas used to force install the correct browser, driver, and updated selenium to match. This caused specific quirks in ordering of steps that had to be adhered to. With this conversion, the sources are now setup to take in the needed paths by an environment variable. This allows theinstall-depsaction to install the browsers as they are an OS level dependency to run. Then the paths are passed to the test runs and utilized. Bypassing the need to run certain installs in a specific order.Verification
The original code in the PR was running on every push to get a test run going. This run was from the commit before the final push which turned off the
on: pushexecution. This shows the workflow successfully running with all nightly tests passing.Refs: #4912