Skip to content

Infrastructure: Fix failing npm run regression-report script#2291

Merged
mcking65 merged 8 commits intomainfrom
bocoup/fix-issue-959
Dec 12, 2023
Merged

Infrastructure: Fix failing npm run regression-report script#2291
mcking65 merged 8 commits intomainfrom
bocoup/fix-issue-959

Conversation

@howard-e
Copy link
Copy Markdown
Contributor

@howard-e howard-e commented Apr 13, 2022 •
edited by michael-n-cooper-bot
Loading

Address comments in #959.

cc @nschonni

WAI Preview Link

@howard-e howard-e added the regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo label Apr 13, 2022
@howard-e howard-e requested a review from mcking65 April 13, 2022 20:50
@howard-e howard-e force-pushed the bocoup/fix-issue-959 branch from b27c712 to dd13ac1 Compare April 13, 2022 20:59
@howard-e howard-e mentioned this pull request Jul 7, 2022
Merged
@howard-e howard-e force-pushed the bocoup/fix-issue-959 branch from dd13ac1 to 373ef16 Compare July 7, 2022 14:29
@mcking65 mcking65 mentioned this pull request Oct 15, 2023
Closed
@howard-e howard-e force-pushed the bocoup/fix-issue-959 branch from 373ef16 to 6c119fc Compare November 1, 2023 13:49
@howard-e
Copy link
Copy Markdown
Contributor Author

howard-e commented Nov 1, 2023

@mcking65 this has been updated following #2837.

The result of npm run regression-report now gives me: 
#### Regression test coverage:


#### Examples without any regression tests:

None found.


#### Examples missing some regression tests:

- dialog-modal/examples/datepicker-dialog.html:
   - textbox-aria-describedby
- listbox/examples/listbox-collapsible.html:
   - key-tab
- listbox/examples/listbox-grouped.html:
   - key-tab
   - key-character
- listbox/examples/listbox-rearrangeable.html:
   - key-tab
   - key-character
- listbox/examples/listbox-scrollable.html:
   - key-tab
   - key-character
- menu-button/examples/menu-button-actions-active-descendant.html:
   - menu-up-arrow
   - menu-down-arrow
   - menu-home
   - menu-character
- menu-button/examples/menu-button-links.html:
   - menu-enter
- menubar/examples/menubar-navigation.html:
   - menubar-menuitem-aria-current
- slider/examples/slider-color-viewer.html:
   - group-aria-labelledby
- slider/examples/slider-temperature.html:
   - aria-valuetext
- switch/examples/switch.html:
   - key-tab
- toolbar/examples/toolbar.html:
   - toolbar-spinbutton-role


#### Example pages with Keyboard or Attribute table rows that do not have data-test-ids:

- alertdialog/examples/alertdialog.html
   - "Keyboard Support" table(s):
      - Shift + Tab
      - Command + S
   - "Attributes" table(s):
      - alert
      - aria-disabled="true"

#### SUMMARY:

  58 example pages found.
  0 example pages have no regression tests.
  12 example pages are missing approximately 18 out of approximately 870 tests.

ERROR - missing tests:

  Please write missing tests for this report to pass.

@howard-e howard-e added the Infrastructure Related to maintaining task force and repo operations, processes, systems, documentation label Nov 1, 2023
This was linked to issues Nov 1, 2023
Update regression test report to work with new repo structure #2837
Closed
Regression test coverage report fials when run locally #959
Closed
@howard-e howard-e requested review from mcking65 and removed request for mcking65 November 1, 2023 16:01
@howard-e howard-e force-pushed the bocoup/fix-issue-959 branch from e2789ed to 6a83aac Compare November 2, 2023 13:33
@howard-e howard-e requested a review from a11ydoer November 2, 2023 13:55
@a11ydoer
Copy link
Copy Markdown
Contributor

a11ydoer commented Nov 2, 2023

@howard-e it may be the separate issue but I have error of missing script:"regression-report" when I did npm run regression-report. When I check the available list of script using "npm run", fmt, lint, and boostrap were only available. I am mentioning @spectranaut in case she can help.

@howard-e
Copy link
Copy Markdown
Contributor Author

howard-e commented Nov 2, 2023

@a11ydoer when I do npm run, I get the following:

npm run result 
❯ npm run
Lifecycle scripts included in aria-practices@0.0.0:
  test
    npm run lint && npm run regression

available via `npm run-script`:
  fix
    npm run lint:es -- --fix && stylelint --fix  "**/*.css" && prettier --write "**/*.css"
  htmlhint
    htmlhint "**/*.html" "**/*.template" --ignore "common/**/*.html" --format unix
  lint
    npm run lint:es && npm run lint:css && npm run lint:html && npm run lint:spelling
  lint:css
    stylelint "**/*.css" && prettier --check "**/*.css"
  lint:es
    eslint . --report-unused-disable-directives
  lint:js
    npm run lint:es
  lint:html
    npm run htmlhint && npm run vnu-jar
  lint:spelling
    cspell "**/*.*"
  link-checker
    node ./scripts/link-checker.js
  reference-tables
    node scripts/reference-tables.js
  regression
    ava --timeout=1m
  regression-report
    node test/util/report
  vnu-jar
    java -jar node_modules/vnu-jar/build/dist/vnu.jar --filterfile .vnurc --no-langdetect --skip-non-html content/
  create-gh-project
    node ./scripts/create-gh-project.js
  prepare
    husky install

I'm not familiar with the fmt and bootstrap commands in this repo. Is it possible you may have ran that command in a separate repo's project?

@a11ydoer
Copy link
Copy Markdown
Contributor

a11ydoer commented Nov 2, 2023 •
edited
Loading

@howard-e sorry. you are right. this is downside of multi-tasking, not focusing on one thing. I was using old repo not updated one.

a11ydoer
a11ydoer approved these changes Nov 3, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@a11ydoer a11ydoer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"npm run regression-report" is working due to directory path update.

test/util/report.js
const spawn = require('cross-spawn');

const examplePath = path.resolve(__dirname, '..', '..', 'examples');
const examplePath = path.resolve(__dirname, '..', '..', 'content', 'patterns');
Copy link
Copy Markdown
Contributor

@a11ydoer a11ydoer Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@howard-e yes, updated path info was what I needed to run regression test. Thanks. I will also check other changes.

Copy link
Copy Markdown
Contributor

@a11ydoer a11ydoer Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@howard-e I confirmed that the path update here works. Thanks again!

@mcking65
Copy link
Copy Markdown
Contributor

mcking65 commented Dec 11, 2023

@howard-e

I am getting a runtime error as follows.

npm run regression-report

aria-practices@0.0.0 regression-report
node test/util/report
C:\Users\mck\git\APGRepo\main\test\util\report.js:208
    exampleCoverage[example].missingTests.delete(matchResults[2]);
                             ^
TypeError: Cannot read properties of undefined (reading 'missingTests')
    at getRegressionTestCoverage (C:\Users\mck\git\APGRepo\main\test\util\report.js:208:30)
    at Object.<anonymous> (C:\Users\mck\git\APGRepo\main\test\util\report.js:217:1)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47
Node.js v18.12.0

@howard-e
Copy link
Copy Markdown
Contributor Author

howard-e commented Dec 11, 2023

@howard-e

I am getting a runtime error as follows.

@mcking65 Oops! Some cross-platform support (for Windows) was missed there. 0423ad7 should address this.

mcking65
mcking65 approved these changes Dec 12, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@mcking65 mcking65 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @howard-e! This is now working.

@mcking65 mcking65 merged commit aa6307a into main Dec 12, 2023
@mcking65 mcking65 deleted the bocoup/fix-issue-959 branch December 12, 2023 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcking65 mcking65 mcking65 approved these changes

@a11ydoer a11ydoer a11ydoer approved these changes

Assignees

No one assigned

Labels

Infrastructure Related to maintaining task force and repo operations, processes, systems, documentation regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Update regression test report to work with new repo structure Regression test coverage report fials when run locally

3 participants

@howard-e @a11ydoer @mcking65