core(audits): add handling of 'incomplete' results from axe-core#10072
core(audits): add handling of 'incomplete' results from axe-core#10072patrickhulce merged 68 commits into
Conversation
|
Resolves #10056 (comment) (cc @patrickhulce) Unit tests pass locally, but Travis seems unreliable for PRs at the moment. I've opened an experimental improvement in #10075. |
This reverts commit e910079.
|
@patrickhulce Any chance you could reset the Travis build cache for this PR? Doesn't look like I have permissions to, and I think that's all that's left to get this passing (I was quietly hoping that merging master might invalidate the cache, but no dice so far) |
|
Just kicked a new one off without the cache, we'll see if it does the trick! |
patrickhulce
left a comment
There was a problem hiding this comment.
LGTM % types
awesome job @jayaddison!
|
@connorjclark any more thoughts? |
|
@patrickhulce @connorjclark One thing I was wondering about - the audit I've opened an upstream ticket in axe-core about it (linked in this PR's history) to see if that can be addressed, but was wondering if it's likely to cause any issues/problems in the meantime? |
It will be a slight regression in the available data, but I don't expect it to be too large of a problem as long as we get this in to 6.0. There's also not much we can do about it other than not upgrade or wait for axe to fix it, so it shouldn't be a blocker IMO. |
|
Small update: |
|
@paulirish does this require further discussion or does the current state of a review requested from @connorjclark work? @connorjclark if you'd prefer deferring to my review that's fine too, just wanted to give you the opportunity since you reviewed earlier. |
|
Thanks a ton @patrickhulce @connorjclark ! |
Summary
As of axe-core@3.4.0, the
form-field-multiple-labelsaudit will no longer indicate a strict failure, but instead returns anincompleteresult.As per the axe-core docs, an
incompleteresult indicates either an error executing a rule or an accessibility result which requires manual investigation.This changeset handles the former error cases by transforming the
incompleteresult into alighthouseaudit error result.Non-error
incompleteresults are considered failure cases.Related Issues/PRs
Additional background and context is available in #10056, particularly this comment.