Automated axe testing by myasonik · Pull Request #2569 · elastic/eui

myasonik · 2019-11-27T01:15:50Z

Summary

Closes #2231 and introduces automated axe testing!

Currently only testing the guidelines so the capability can be merged. Adding more pages over time can become a ticket an a few pages can be added at a time when people have spare cycles. (For reference, there are currently 84 errors found when running it across all pages. This does not count color contrast rules which had to be disabled due to their quantity...)

Run npm run start-test-server-and-a11y-test to run the tests locally.

TODO

How to make this run in CI to block merge? (Added to test script)
Make an issue for adding all the pages ([Meta][Docs] Add automated tests for all pages #2679)
What sort of documentation does this need? (Added wiki page on it)
On the Colors Guidelines there's a duplicate ID error in the range slider that I can't track down... (@thompsongl put up Alter id on EuiFormRow wrapping element #2588 to address this)

Checklist

- [ ] Checked in dark mode
- [ ] Checked in mobile
- [ ] Checked in IE11 and Firefox
- [ ] Props have proper autodocs
~~- [ ] Added documentation examples~~
- [ ] Added or updated jest tests
~~- [ ] Checked for breaking changes and labeled appropriately~~
~~- [ ] Checked for accessibility including keyboard-only and screenreader modes~~
~~- [ ] A changelog entry exists and is marked appropriately~~

thompsongl · 2019-12-03T14:53:39Z

On the Colors Guidelines there's a duplicate ID error in the range slider that I can't track down...

I'll take a look today

thompsongl · 2019-12-03T16:11:01Z

^ #2588 has what I think is the right way to fix the duplicate id error, but check my math there.

chandlerprall · 2019-12-11T20:40:11Z

Overall, these changes look good. I haven't reviewed a11y-testing.js or the readme changes in depth yet, but everything looks good functionally. A blocker for this moving forward is CI's failure when running

20:21:56 > @elastic/eui@17.0.0 test-a11y /app
20:21:56 > node ./scripts/a11y-testing
20:21:56 
20:21:57 (node:559) UnhandledPromiseRejectionWarning: Error: Failed to launch chrome!
20:21:57 /app/node_modules/puppeteer/.local-chromium/linux-706915/chrome-linux/chrome: error while loading shared libraries: libX11-xcb.so.1: cannot open shared object file: No such file or directory
20:21:57 
20:21:57 
20:21:57 TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md
20:21:57 
20:21:57     at onClose (/app/node_modules/puppeteer/lib/Launcher.js:348:14)
20:21:57     at Interface.helper.addEventListener (/app/node_modules/puppeteer/lib/Launcher.js:337:50)
20:21:57     at Interface.emit (events.js:203:15)
20:21:57     at Interface.close (readline.js:397:8)
20:21:57     at Socket.onend (readline.js:173:10)
20:21:57     at Socket.emit (events.js:203:15)
20:21:57     at endReadableNT (_stream_readable.js:1143:12)
20:21:57     at process._tickCallback (internal/process/next_tick.js:63:19)
20:21:57 (node:559) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
20:21:57 (node:559) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

which demonstrates both a failure to run puppeteer and that the error didn't fail the CI job, instead it crashed "silently".

thompsongl · 2020-01-22T23:29:33Z

Just an update: I've been working on some Docker and script changes to get the necessary deps into EUI CI runs. Learning some things and getting closer to understanding what the limitations are. I'll continue to with infra to get this going.

This reverts commit f0c8f9f.

thompsongl · 2020-02-07T17:50:01Z

jenkins test this

^ Last commit works in Docker locally, so I suspect some user/permission thing with the Jenkins setup

kibanamachine · 2020-03-02T21:17:29Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-02T21:31:03Z

jenkins test this

Going trigger a few more builds to see about stability.

The recent change is to run Chromium without a sandbox, which is generally only acceptable in cases like this where we fully own the content being crawled.

kibanamachine · 2020-03-02T21:51:57Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-02T21:56:07Z

jenkins test this

kibanamachine · 2020-03-02T22:15:01Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-02T22:24:40Z

jenkins test this

kibanamachine · 2020-03-02T22:43:09Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-02T23:34:04Z

jenkins test this

kibanamachine · 2020-03-02T23:53:20Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-03T15:12:13Z

jenkins test this

kibanamachine · 2020-03-03T15:30:22Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-03T16:10:49Z

jenkins test this

kibanamachine · 2020-03-03T16:29:19Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-03T16:36:50Z

jenkins test this

kibanamachine · 2020-03-03T16:55:02Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-03T17:15:27Z

jenkins test this

kibanamachine · 2020-03-03T17:32:50Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-03T17:59:21Z

I'm going to call this "stable enough" and do some clean up 🎉

kibanamachine · 2020-03-03T20:39:55Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

thompsongl · 2020-03-03T20:46:24Z

Ready for your review, @chandlerprall

package.json

chandlerprall

Changes LGreatTM, pulled & tested locally in both a passing & error states, with both the automated test server on port 9999 & against the local development server.

thompsongl · 2020-03-05T19:31:24Z

LGreatTM

😅

kibanamachine · 2020-03-05T19:57:07Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_2569/

PoC of automated axe testing

6dffcd4

myasonik requested a review from chandlerprall November 27, 2019 01:15

fixing a11y bugs guidelines pages

9ff667c

thompsongl mentioned this pull request Dec 3, 2019

Alter id on EuiFormRow wrapping element #2588

Merged

3 tasks

Michail Yasonik added 4 commits December 3, 2019 11:23

Adds a11y tests to test command

c4b5c18

adding a11y script

2d8d4d7

adding docs

d2ab39c

Merge remote-tracking branch 'upstream/master' into axe-testing

6b594b6

myasonik marked this pull request as ready for review December 5, 2019 19:20

fixing logging for yarn

aadee63

catching errors in puppeteer setup to fail builds

317366d

myasonik mentioned this pull request Dec 17, 2019

[Meta][Docs] Add automated tests for all pages #2679

Closed

26 tasks

myasonik force-pushed the axe-testing branch 2 times, most recently from e7387c5 to 38b1e93 Compare December 18, 2019 22:11

experimenting with docker

338999f

myasonik force-pushed the axe-testing branch from 38b1e93 to 338999f Compare December 19, 2019 02:44

Michail Yasonik added 2 commits December 19, 2019 16:55

retest

8a16eb2

one step forward

6021961

thompsongl mentioned this pull request Jan 27, 2020

[Meta] EUI Infra Improvements #2799

Closed

14 tasks

thompsongl added 6 commits February 6, 2020 16:21

Merge branch 'master' into axe-testing

f984615

deps

949c708

trial run

3c7a06a

trial run

f0c8f9f

Revert "trial run"

34c786e

This reverts commit f0c8f9f.

user

396c970

thompsongl added 2 commits March 2, 2020 14:38

Merge branch 'master' into axe-testing

d10e0b4

sandbox trial

e0bad11

thompsongl added 2 commits March 3, 2020 13:23

Merge branch 'master' into axe-testing

76b800e

move to script file

1ceef4e

chandlerprall reviewed Mar 4, 2020

View reviewed changes

package.json Show resolved Hide resolved

chandlerprall approved these changes Mar 5, 2020

View reviewed changes

Merge branch 'master' into axe-testing

6363858

thompsongl merged commit 68299f4 into elastic:master Mar 5, 2020

Conversation

myasonik commented Nov 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

TODO

Checklist

Uh oh!

thompsongl commented Dec 3, 2019

Uh oh!

thompsongl commented Dec 3, 2019

Uh oh!

chandlerprall commented Dec 11, 2019

Uh oh!

thompsongl commented Jan 22, 2020

Uh oh!

thompsongl commented Feb 7, 2020

Uh oh!

kibanamachine commented Mar 2, 2020

Uh oh!

thompsongl commented Mar 2, 2020

Uh oh!

kibanamachine commented Mar 2, 2020

Uh oh!

thompsongl commented Mar 2, 2020

Uh oh!

kibanamachine commented Mar 2, 2020

Uh oh!

thompsongl commented Mar 2, 2020

Uh oh!

kibanamachine commented Mar 2, 2020

Uh oh!

thompsongl commented Mar 2, 2020

Uh oh!

kibanamachine commented Mar 2, 2020

Uh oh!

thompsongl commented Mar 3, 2020

Uh oh!

kibanamachine commented Mar 3, 2020

Uh oh!

thompsongl commented Mar 3, 2020

Uh oh!

kibanamachine commented Mar 3, 2020

Uh oh!

thompsongl commented Mar 3, 2020

Uh oh!

kibanamachine commented Mar 3, 2020

Uh oh!

thompsongl commented Mar 3, 2020

Uh oh!

kibanamachine commented Mar 3, 2020

Uh oh!

thompsongl commented Mar 3, 2020

Uh oh!

kibanamachine commented Mar 3, 2020

Uh oh!

thompsongl commented Mar 3, 2020

Uh oh!

Uh oh!

chandlerprall left a comment

Choose a reason for hiding this comment

Uh oh!

thompsongl commented Mar 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kibanamachine commented Mar 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

myasonik commented Nov 27, 2019 •

edited

Loading

thompsongl commented Mar 5, 2020 •

edited

Loading