Skip to content

Tests reusing samples must have unique titles.#14265

Merged
feerrenrut merged 2 commits into
masterfrom
systest-ensureUniqueSamples
Oct 21, 2022
Merged

Tests reusing samples must have unique titles.#14265
feerrenrut merged 2 commits into
masterfrom
systest-ensureUniqueSamples

Conversation

@feerrenrut

@feerrenrut feerrenrut commented Oct 17, 2022

Copy link
Copy Markdown
Contributor

Link to issue number:

Splitting up PR #14054

Summary of the issue:

When the same sample is used in multiple tests, the title for the app (chrome / notepad) may be identical.
The tests use the app title to determine when the test sample has opened.
When the titles are not unique, the test may start with the app already in an unexpected state, it may also get interrupted by the new app instance opening.

Description of user facing changes

None

Description of development approach

Use a timestamp as well as the sample contents to create a unique title.

Testing strategy:

Run on appveyor.

Known issues with pull request:

None

Change log entries:

None

Code Review Checklist:

  • Pull Request description:
    • description is up to date
    • change log entries
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • API is compatible with existing add-ons.
  • Documentation:
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • Security precautions taken.

@feerrenrut
Handle sample re-use in system tests
548200f 
Some tests use the same testcase/sample with different NVDA settings or
a different test procedure.
To ensure the correct document is focused, append a timestamp comment
to the testCase HTML, giving a unique hash for each test.
@feerrenrut
ensure notepad testcases have a unique title
ef1b220
@AppVeyorBot

This comment was marked as outdated.

Sign in to view
@feerrenrut

feerrenrut commented Oct 19, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Prior build test failure:
symbolPronunciationTests Test TableHeaders is a test that runs in Chrome.

From the Robot Framework log:

22:17:04.946	DEBUG	Found window (HWND: 262680) (title: NVDA Browser Test Case (726508196) - Google Chrome)	
22:17:04.948	INFO	get_last_speech_index	
22:17:04.959	INFO	emulateKeyPress ('NVDA+t',)	
22:17:05.584	INFO	wait_for_specific_speech ('NVDA Browser Test Case (726508196)',) {'afterIndex': 1}	
22:17:10.584	FAIL	Specific speech did not occur before timeout: NVDA Browser Test Case (726508196)
See NVDA log for dump of all speech.

Note, the test waited 5 seconds for the title to be reported.

The screenshot shows Chrome, tab with title "NVDA Browser Test Case (726508..." visible and a "Restore pages" bubble window:

Chrome didn't shutdown correctly

Chrome crashing is not good, but in this case, it seems to have interfered with the subsequent test.
This should be fixed by #14262.

The tests are queued to run again.

@feerrenrut feerrenrut marked this pull request as ready for review October 20, 2022 08:17
@feerrenrut feerrenrut requested a review from a team as a code owner October 20, 2022 08:17
@feerrenrut feerrenrut requested a review from seanbudd October 20, 2022 08:17
@feerrenrut feerrenrut merged commit d0bc25d into master Oct 21, 2022
@feerrenrut feerrenrut deleted the systest-ensureUniqueSamples branch October 21, 2022 02:38
@nvaccessAuto nvaccessAuto added this to the 2023.1 milestone Oct 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@seanbudd seanbudd seanbudd approved these changes

Assignees

No one assigned

Labels

component/system-testing

Projects

None yet

Milestone

2023.1

Development

Successfully merging this pull request may close these issues.

4 participants

@feerrenrut @AppVeyorBot @seanbudd @nvaccessAuto