Skip to content

Investigate flakey system tests #18597

@seanbudd

Description

@seanbudd

Related issues, PRs or discussions

What is the current state of the codebase?

A stress test was run on our system tests, running each test 40 times:

The following failures occurred:
Generic:

  • 3x Speech did not finish before timeout
  • Startup/shutdown:
    • 5x: Ensure NVDA restarts on crash: Old NVDA is still running.
  • Symbols
    • 3x: symbolInSpeechUI :: Ensure symbols aren't substituted within NVDA. Unable to focus Notepad.
  • Chrome:
    • 1x: ARIA details noVbuf with free review and nav. Timed out waiting for key to be processed
    • 2x: [ WARN ] Unable to close tab, Chrome window not initialised correctly.

With 14 separate test failures in 40 runs, its expected that the average PR build will fail roughly 1/3 of the time

Why are changes required?

Tests should pass regularly without intermittent failures

What technical changes are required?

There might be an actual issue with NVDA crash handling.

Since only the first test for notepad is failing, starting notepad prior to testing might help.

Speech not finishing before timeout usually involves a loss of focus or freeze, but it could just need a timeout increase.

It seems like a CMD shell spawned by GitHub actions can hijack the focus from notepad and Chrome

Are the proposed technical changes API breaking?

No

Are there potential risks or issues with the proposed implementation?

No

Metadata

Metadata

Assignees

No one assigned

    Labels

    component/system-testingp3https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#prioritytriagedHas been triaged, issue is waiting for implementation.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions