Skip to content

UIA handler/notification event: handle elements without window handle property set such as window visual state announcements from File Explorer/Windows shell#18229

Merged
SaschaCowley merged 24 commits into
nvaccess:masterfrom
josephsl:i18220UIANotificationNoWindowHandle
Jun 24, 2025

Conversation

@josephsl

@josephsl josephsl commented Jun 7, 2025

Copy link
Copy Markdown
Contributor

Link to issue number:

Closes #18220
Closes #17841

Summary of the issue:

NVDA does not announce window visual states such as restore and snap when Windows+arrow keys are pressed.

Description of user facing changes:

NVDA will announce window visual states (for the foreground window) when Windows+arrow keys are pressed.

Description of developer facing changes:

Window state notifications are raised by a Windows shell (File Explorer) element without a native window handle. Therefore, allow notifications to be processed if the app module says yes, and if so, use foreground window handle as "window handle" for the element (because UIA NVDA objects require a window handle during construction even though all that will be done is process events).

Description of development approach:

In UIA handler/notification event handler:

  • If native window handle is not found (including traversing the UIA tree back to the root element), ask the app module for the process (where the element lives) if notification event should be processed.
  • If the app module says yes, set foreground window handle as "window handle" for the UIA element.

Base app module:

  • Added a new method, "shouldProcessUIANotificationEventNoWindowHandle" to inform UIA handler if UIA notifications for elements without native window handle should be processed.
  • The new method will return False by default in line with previous behavior of dropping notification events for elements without window handle property set.

File Explorer app module:

  • Say "yes" to "shouldProcessUIANotificationEventNoWindowHandle" call if window snap activity Id is being handled.
  • Added app module version of UIA notification event handler to announce window visual state/snap notification from everywhere.

Testing strategy:

Manual testing: from any app window (such as a web browser), press Windows+arrw keys to snap windows to part of the screen or press Windows+up arrow to maximize the foreground window.

Prior to this PR: no window state announcement.
After this PR: NVDA will announce window snap notification.

Known issues with pull request:

While the foreground window handle is an obvious choice (as the nanouncemenet pertians to window visual state changes), there might be a better way such as a custom UIA property. Ideally, window handle should be exposed by this shell element and hopefully Microsoft can resolve this. Note that File Explorer is not the only app with this issue - Windows 11 Voice Access exhibits the same problem (#16862).

Code Review Checklist:

  • Documentation:
    • Change log entry
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • API is compatible with existing add-ons.
  • Security precautions taken.

@coderabbitai summary

josephsl added 7 commits June 7, 2025 05:20
…from elements without window handles. Re nvaccess#18220.

UIA notification event fails if UIA element has no native window handle. However, some of these elements report system information such as window visual state (restore/maximize/snap) yet do not set native window handle value. Therefore, ask app modules if notifications from these elements should be accepted (notable case: File Explorer/window visual states).
…onEventNoWindowHandle method to determine if notifications should be received from elements without hwnd set. Re nvaccess#18220.

Similar to UIA propewrty change event handler: add a method in the base app module to say yes or no if asked to process UIA notification events for elements (senders) without window handle defined. Method signature is same as UIA notification event except arguments other than sender (element) are keyword arguments - not all app modules may wish to work with all UIA notification event arguments. By default, this method will return False for compatibility with previous NVDA behavior of dropping notification events without window handle set.
…as 'widow handle' if native window handle is not set for an element. Re nvaccess#18220.

UIA NVDA objects need window handle as part of its construction, including while processing events. However, some elements with important notifications (notably window visual state notifications) do not set native window handle property. Therefore, set foreground window handle as the element's 'window handle' to allow notification event to propagate across handlers (global plugins, app modules, NVDA object).
…from everywhere. Re nvaccess#17841, nvaccess#18175.

The element that reports window visual state changes (restore/masimize/minimize/snap) does not define a native window handle property. Therefore, let File Explorer app module say 'yes' when asked to process UIA notification event from this element. Note that this is not the only time a 'no window handle notification' is used - the other case is input layout switch notifications (say, changing between input languages), thus limit processing to snap anouncements.
@josephsl josephsl requested a review from a team as a code owner June 7, 2025 12:18
@josephsl josephsl requested a review from SaschaCowley June 7, 2025 12:18
@josephsl josephsl changed the title UIA handler/notification event: handle elmenets without window handles such as window visual state announcements UIA handler/notification event: handle elements without window handle property set such as window visual state announcements from File Explorer/Windows shell Jun 7, 2025
Comment thread source/appModules/explorer.py Outdated
Comment thread source/UIAHandler/__init__.py Outdated
josephsl added 4 commits June 8, 2025 10:03
…NOtificationEventNoWindowHandle -> shouldProcessUIANotificationEvent (generalizability). Re nvaccess#18220, nvaccess#17841.

Rename the app module method to make it more generalizable (including handling events from elements with no window handle set).
…ess#18220.

Nearest window handle getter from UIA handler can say 'None' if window is not found, and calling bool(None) gives False. Therefore, use the boolean transformation of the nearest window handle call as base implementation for 'shouldProcessUIANotificationEvent' method.
…dow handle can report notifications only if app modules say so). Re nvaccess#17841, nvaccess#18220
Comment thread source/appModules/explorer.py Outdated
Comment thread source/appModules/explorer.py Outdated
Comment thread source/appModules/explorer.py Outdated
…ificationEvent/event_UIA_notification instead of using kwargs dictionary. Re nvaccess#18220.

Reviewed by Leonard de Ruijter: spell out argument names for shouldProcessUIANotificationEvent and event_UIA_notification to make the code more readable.
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit a3c8d70668

@seanbudd seanbudd requested a review from michaelDCurran June 10, 2025 00:10
@seanbudd seanbudd added the blocked/needs-product-decision A product decision needs to be made. Decisions about NVDA UX or supported use-cases. label Jun 10, 2025
Comment thread user_docs/en/changes.md Outdated
josephsl and others added 2 commits June 10, 2025 04:58
NOted by Cyrille Bougot: add grave accent (1) to emphasize hotkeys.

Co-authored-by: Cyrille Bougot <cyrille.bougot2@laposte.net>
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit d282f4cfff

@SaschaCowley SaschaCowley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that @michaelDCurran has been requested to review this as well due to my inexperience with UIA

Comment thread source/UIAHandler/__init__.py Outdated
Comment thread source/appModuleHandler.py Outdated
Comment thread source/appModuleHandler.py Outdated
Comment thread source/appModuleHandler.py
Comment thread source/appModules/explorer.py Outdated
josephsl added 3 commits June 14, 2025 09:21
…ment in notification event handler. Re nvaccess#18220.

Note from Sascha Cowley (NV ACcess): add type hints please (do this in app modules by importing UIA client from COM interfaces).
josephsl added 2 commits June 15, 2025 06:06
…ionEvent' method. Re nvaccess#18220.

Note from Sascha Cowley (NV Access): add parameter hints to notification processing event to improve docstring clarity.
… to UIA elements without window handles. Re nvaccess#18220.

NOte from reviewers: would it be better to use desktop window handle rather than foreground handle? The issue: some UIA elements do not connect back to the desktop through UIA tree traversal. The foreground window handle makes sense when handling the primary case of window visual state announcement, but there are other places where UIA notification events without window handles (at least from NVDA's perspective) will be seen such as Voice Access (Windows 11). Therefore, switch to using desktop window handle to communicate the need to handle notification events more generically.
@seanbudd seanbudd added conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review. and removed blocked/needs-product-decision A product decision needs to be made. Decisions about NVDA UX or supported use-cases. labels Jun 16, 2025

@michaelDCurran michaelDCurran left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, at least on the approach.

@josephsl

josephsl commented Jun 18, 2025 via email

Copy link
Copy Markdown
Contributor Author

Comment thread source/appModuleHandler.py
Comment thread source/appModuleHandler.py
Comment thread source/appModuleHandler.py Outdated
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit 4465872125

@SaschaCowley SaschaCowley merged commit f6844c3 into nvaccess:master Jun 24, 2025
16 of 17 checks passed
@github-actions github-actions Bot added this to the 2025.2 milestone Jun 24, 2025
@josephsl josephsl deleted the i18220UIANotificationNoWindowHandle branch June 25, 2025 00:21
seanbudd pushed a commit that referenced this pull request Jun 25, 2025
… NotificationProcessing -> notificationProcessing (argument name consistency) (#18319)

Quick follow-up to #18229

Summary of the issue:

UIA notification argument names are inconsistent across UIA handler, app modules, and UIA NVDA objects.
Description of user facing changes:

None
Description of developer facing changes:

None, as the app module API was just merged.
Description of development approach:

Change the follwoing argument names in app module version of shouldProcessUIANotificationEvent:

    NotificationKind -> notificationKind
    NotificationProcessing -> notificationProcessing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review.

Projects

None yet

7 participants