chore: Test against ARIA practices by WilcoFiers · Pull Request #3144 · dequelabs/axe-core

WilcoFiers · 2021-09-02T12:20:11Z

One of the more common reasons for reported false positives comes from axe-core failing something recommended in the ARIA practices. This PR adds a test script will have axe-core running against the ARIA practices.

Set up basic test script
Open issues for problems discovered in the test:apg
Lock the APG dependency to a fix commit for stable testing
Set up CI, including a nightly test where we're running against the latest on #main

Closes issue: #2918

CLAassistant · 2021-09-02T12:20:17Z

All committers have signed the CLA.

WilcoFiers · 2021-09-03T10:12:36Z

PR is blocked until the aria-practices removes husky from its postinstall script. I opened a PR for that: w3c/aria-practices#2033

stephenmathieson · 2021-11-05T18:24:04Z

.circleci/config.yml

    jobs:
      - dependencies_unix
-      - test_nightly:
+      - test_nightly_chrome:


How will we know if/when this fails? Is there someone responsible for checking CI logs daily?

Yeah, those get kicked out to our #attest-build channel on Slack.

I also get emails every night about the run and see if they pass/fail

test/aria-practices/apg.spec.js

WilcoFiers · 2021-11-08T10:09:28Z

.circleci/config.yml

 unix_box: &unix_box
  docker:
-    - image: circleci/node:10-browsers
+    - image: circleci/node:16-browsers


So that we get access to Chrome 95

Just an fyi, circle is changing how they tag their images (should be cimg/node:16.13-browsers https://circleci.com/developer/images/image/cimg/node). I'll update the tag in another pr because we can also specify using lts-browsers which I think would be good to test on our nightly builds

straker · 2021-11-08T16:00:28Z

.circleci/config.yml

 unix_box: &unix_box
  docker:
-    - image: circleci/node:10-browsers
+    - image: circleci/node:16-browsers


Just an fyi, circle is changing how they tag their images (should be cimg/node:16.13-browsers https://circleci.com/developer/images/image/cimg/node). I'll update the tag in another pr because we can also specify using lts-browsers which I think would be good to test on our nightly builds

straker · 2021-11-08T16:02:53Z

.circleci/config.yml

    jobs:
      - dependencies_unix
-      - test_nightly:
+      - test_nightly_chrome:


I also get emails every night about the run and see if they pass/fail

straker · 2021-11-08T16:04:28Z

package.json

    "fmt": "prettier --write .*.json *.{json,md,js} **/*.ts './{.circleci,.github,build,doc,lib,locales,test,typings}/**/*.{json,md,js,ts,html}'"
  },
  "devDependencies": {
+    "@axe-core/webdriverjs": "^4.2.2",


This is a circular dependency, will this cause any problems (probably not, but want to make absolutely sure)?

Don't think so. It's a dev dependency.

straker · 2021-11-08T16:13:28Z

package.json

    "@babel/preset-env": "^7.5.4",
    "@deque/dot": "^1.1.5",
    "act-rules.github.io": "github:act-rules/act-rules.github.io#master",
+    "aria-practices": "github:w3c/aria-practices#f7de7ec3a53534018237f24cb9e610f26c30c367",


How often should we update the commit hash?

I was thinking that if nightly fails, we take another look. Don't think we need to do it more often than that.

straker · 2021-11-08T16:15:27Z

test/aria-practices/apg.spec.js

+  const port = 9515;
+  let driver, server, addr, axeSource;
+  this.timeout(50000);
+  this.retries(3);


Was this failing a lot that it needed retries?

I'm not entirely sure. I did see it failing once or twice for a reason I couldn't figure out. Don't know for sure what the reason of that was. Don't think having this here hurts.

straker · 2021-11-08T16:47:37Z

test/aria-practices/apg.spec.js

+      'region', // dequelabs/axe-core#3260
+      'heading-order', // APG issues
+      'list', // APG issues
+      'scrollable-region-focusable', // w3c/aria-practices#2114


I like these comments a lot. Thanks for adding them.

straker · 2021-11-08T17:13:33Z

test/aria-practices/apg.spec.js

+    await new Promise(r => setTimeout(r, 500));
+    await connectToChromeDriver(port);
+
+    const app = express();


Are any of the test files using absolute paths? If not I think our npm run start command should serve node_modules directly as well, so we might not need express for this.

Fair point, let me see if I can make that work.

test/aria-practices/apg.spec.js

Co-authored-by: Steven Lambert <2433219+straker@users.noreply.github.com>

WilcoFiers · 2021-11-09T10:41:36Z

package.json

    "@babel/preset-env": "^7.5.4",
    "@deque/dot": "^1.1.5",
    "act-rules.github.io": "github:act-rules/act-rules.github.io#master",
+    "aria-practices": "github:w3c/aria-practices#f7de7ec3a53534018237f24cb9e610f26c30c367",


I was thinking that if nightly fails, we take another look. Don't think we need to do it more often than that.

WilcoFiers · 2021-11-09T10:43:38Z

test/aria-practices/apg.spec.js

+  const port = 9515;
+  let driver, server, addr, axeSource;
+  this.timeout(50000);
+  this.retries(3);


I'm not entirely sure. I did see it failing once or twice for a reason I couldn't figure out. Don't know for sure what the reason of that was. Don't think having this here hurts.

WilcoFiers · 2021-11-09T10:45:40Z

test/aria-practices/apg.spec.js

+    await new Promise(r => setTimeout(r, 500));
+    await connectToChromeDriver(port);
+
+    const app = express();


Fair point, let me see if I can make that work.

WilcoFiers · 2021-11-09T10:57:48Z

.circleci/config.yml

          npm run test -- --browsers Chrome,FirefoxNightly

+  # Run the test suite for nightly builds.
+  test_nightly_aria_practices:


We should do this same nightly thing with ACT rules. I'll open that PR when this is merged.

done

chore: Test against ARIA practices

ce1aba6

WilcoFiers requested a review from straker September 2, 2021 12:20

WilcoFiers added 5 commits November 5, 2021 17:16

Merge branch 'develop' into aria-practices2

e70ac6c

test: run APG#main nightly

b7b5fff

chore: sort out linting

430f222

chore: bump CI image

dc3d959

chore: Ready for review

300e0d1

WilcoFiers marked this pull request as ready for review November 5, 2021 18:16

WilcoFiers requested a review from a team as a code owner November 5, 2021 18:16

stephenmathieson reviewed Nov 5, 2021

View reviewed changes

test/aria-practices/apg.spec.js Show resolved Hide resolved

stephenmathieson reviewed Nov 5, 2021

View reviewed changes

test/aria-practices/apg.spec.js Outdated Show resolved Hide resolved

WilcoFiers force-pushed the aria-practices2 branch from 61891e9 to 300e0d1 Compare November 8, 2021 09:46

chore: tweaks

a4510be

WilcoFiers commented Nov 8, 2021

View reviewed changes

straker reviewed Nov 8, 2021

View reviewed changes

WilcoFiers added 2 commits November 9, 2021 12:00

chore: remove express

c605363

chore: undo unintended change

87bf8dc

straker previously requested changes Nov 9, 2021

View reviewed changes

test/aria-practices/apg.spec.js Outdated Show resolved Hide resolved

test/aria-practices/apg.spec.js Outdated Show resolved Hide resolved

Apply suggestions from code review

e6e7ed1

Co-authored-by: Steven Lambert <2433219+straker@users.noreply.github.com>

WilcoFiers commented Nov 9, 2021

View reviewed changes

straker approved these changes Nov 9, 2021

View reviewed changes

straker merged commit ed9ef99 into develop Nov 9, 2021

straker deleted the aria-practices2 branch November 9, 2021 18:27

Malvoz mentioned this pull request Jan 9, 2022

Test axe-core against ARIA practices #2918

Closed

Conversation

WilcoFiers commented Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WilcoFiers commented Sep 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

straker Nov 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WilcoFiers commented Sep 2, 2021 •

edited

Loading

CLAassistant commented Sep 2, 2021 •

edited

Loading

straker Nov 8, 2021 •

edited

Loading