Walkthrough

The changes in this update focus on enhancing NVDA's handling of UIA notification events, particularly for Windows 11 Voice Access. This involves explicitly defining argument types for a function, updating a conditional block for handling native window handles, and ensuring NVDA does not give up on finding a native window handle when it's available through the UIA runtime ID.

Changes

Sequence Diagram(s)

sequenceDiagram
    participant NVDA
    participant UIAHandler
    participant NVDAObjects

    NVDA->>UIAHandler: IUIAutomationNotificationEventHandler_HandleNotificationEvent
    UIAHandler->>NVDAObjects: Create UIA object with windowHandle and UIAElement
    Note right of NVDAObjects: If windowHandle is not found, use UIA runtime ID.
    NVDAObjects->>NVDA: Return UIA object
    NVDA->>User: Announce notification (e.g., "microphone active/sleep")

Assessment against linked issues

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

• PASS: Translation comments check.
• PASS: License check.
• PASS: Unit tests.
• FAIL: Lint check. See test results and lint artifacts for more information.
• PASS: System tests (tags: installer NVDA).
• Build (for testing PR): https://ci.appveyor.com/api/buildjobs/btdivse8eu8c2m5p/artifacts/output/nvda_snapshot_pr16872-32944,6d0b0972.exe
• CI timing (mins):
  INIT 0.0,
  INSTALL_START 1.1,
  INSTALL_END 0.9,
  BUILD_START 0.0,
  BUILD_END 11.7,
  TESTSETUP_START 0.0,
  TESTSETUP_END 0.3,
  TEST_START 0.0,
  TEST_END 18.3,
  FINISH_END 0.2

See test results for failed build of commit 6d0b097280

@coderabbitai is there anything you can say about the assumption that the window handle is part of an UIA runtime id?

@LeonarddeR, UI Automation (UIA) runtime IDs are unique identifiers assigned to automation elements, and their structure can vary depending on the element and the application providing the UIA implementation. Typically, a runtime ID is an array of integers, where the presence of a window handle as part of this array is not guaranteed but can occur, especially for elements associated directly with a window.

In the context of the NVDA changes you're reviewing, the assumption that a window handle could be part of the runtime ID is based on practical observations and specific use cases, such as with Windows 11 Voice Access. However, it's important to note that this is not a universal rule for all UIA elements. The modification made in the PR to check the runtime ID for a window handle when getNearestWindowHandle returns None is a way to ensure more robust handling of UIA elements that do include the window handle in their runtime IDs, particularly when other methods fail to retrieve this handle.

This approach helps in cases where the window handle is crucial for properly associating the UIA element with its corresponding UI context, enhancing NVDA's ability to provide accurate notifications and interactions.

I know that at least accesskit creates a runtime identifier based on its internal node ID. Therefore I don't think the current approach is feasible.
Why is NVDA unable to report notifications for objects without a window handle?

Hi,

Okay, let's see what Code Rabbit says.

@coderabbitai analyze other event handlers in UIA handler module and tell us why a window handle is a must.

Postscript: I'm doing this to train the bot and point out possible inaccuracies from AI summaries. Thanks.

@josephsl, after analyzing the event handlers in the source/UIAHandler module, it is evident that window handles play a crucial role in the accurate handling of UIA events. Here are the key points:

1. Associating UIA Elements with UI Context:

   • Event handlers like IUIAutomationEventHandler_HandleAutomationEvent and IUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent use window handles to associate UIA elements with their corresponding UI context. This association is essential for accurately processing events related to the UI elements.

2. Ensuring Accurate Event Handling:

   • Window handles are used to verify whether an event should be accepted or ignored. For instance, in IUIAutomationEventHandler_HandleAutomationEvent, the window handle is checked to determine if the event should be processed (shouldAcceptEvent). This helps in filtering out irrelevant or duplicate events, improving the reliability of event handling.

3. Performance Considerations:

   • Caching window handles can enhance performance by avoiding redundant computations. For example, getNearestWindowHandle caches the window handle to speed up subsequent queries for the same UIA element.

4. Handling Special Cases:

   • In some cases, such as with Windows Defender Application Guard (WDAG), window handles are used to identify and handle remote UIA elements appropriately.

In summary, window handles are essential for ensuring that UIA events are accurately associated with their UI context, processed correctly, and handled efficiently. This is why the presence of a window handle is often checked and utilized in the event handlers within the UIAHandler module.

#!/bin/bash
# Search for the UIA handler module and extract the implementation of event handlers to analyze the use of window handles.
fd "UIAHandler" --exec cat {} \; | grep -A 5 -B 5 "event handler"

#!/bin/bash
# Search for function definitions related to event handlers within the UIA handler module files to understand the use of window handles.
ast-grep --lang python --pattern $'def $_($_, $_, $_, $_, $_) -> $_:\n    $$$\n    # Event handler' --source-path source/UIAHandler

#!/bin/bash
# Correct the syntax to search for function definitions related to event handlers within the UIA handler module files.
ast-grep --lang python --pattern $'def $_($_, $_, $_, $_, $_) -> $_:\n    $$$\n    # Event handler' source/UIAHandler

#!/bin/bash
# Search for all function definitions within the UIA handler module files to identify event handlers and their use of window handles.
ast-grep --lang python --pattern $'def $_($_, $$$):\n    $$$' source/UIAHandler

Learnt from: josephsl
URL: https://github.com/nvaccess/nvda/pull/16872

Timestamp: 2024-07-17T14:24:55.234Z
Learning: Window handles are crucial in the `UIAHandler` module for associating UIA elements with their UI context, ensuring accurate event handling, and improving performance.

Hi,

At least Code Rabbit gave some answers, but that's only part of the picture. It is correct that window handles are needed to identify UIA elements. More importantly, window handles are needed to construct a more accurate view of a UIA element, as without this, UIAHandler.handler.getNearestWindowHandle can fail to notice elements due to COM error while executing walkers. Also, creation of UIA NVDA object ensures consistency with IAccessible/IA2 objects and allows other components such as app modules and default object implementation to handle events, although creation of objects and a chain of events can take some time to complete (performance).

The heart of the issue is inaccurate UIA element (sender) information in apps such as Voice Access when they raise notification event. The fact that another assistive technology (Narrator) must be active to see correct sender information is concerning. While I agree that retrieving window handle through runtime ID is a fragile approach, it is one sure way to identify elements (window handle check was also performed via dev info (NVDA+F1) command).

Thanks.

The heart of the issue is inaccurate UIA element (sender) information in apps such as Voice Access when they raise notification event.

This makes me think about how we handle invalid window handles with WDAG. May be that approach should be generalized instead?

The fact that another assistive technology (Narrator) must be active to see correct sender information is concerning.
Could you elaborate?

Hi,

Generalizing WDAG approach: perhaps we could recycle that approach but to cover background events as well (WDAG is deprecated). For Voice Access, the most effective use case is dictation while not focused on Voice Access interface itself.

Comment on sender info with Narrator active: when you run Voice Access whiel using NVDA with debug logging on (with UIA included), you'll notice that NVDA will log that there is no window handle. However, when Narrator runs, notification is announced and the UIA object is created successfully; when Narrator exits, NVDA reverts to the inablity to obtain window handle.

Thanks.

Hi,

Upon further investigation, while handling notification events, it turns out elements raising this event become "native" while Narrator is running, "non-native" otherwise. Therefore, this issue may require editing native UIA element fetcher (UIA handler) to tell NVDA to treat some elements as native even though it is not seen as such at first. While editing notification event handler could be a useful change, I think the issue lies deeper (this is why I'm not taking Sean's suggested edits yet).

Thanks.

May be it makes sense to contact Microsoft about this?

Hi,

Sure.

I have narrowed the issue to UIAHandler.handler.getNearestWindowHandle - the method that says yes or no about native UIA element depends on nearest window handle getter. In short, tree walker and window handle caching succeeds or fails in Voice Access while Narrator is enabled and disabled, respectively.

There is a related issue that also deals with UIA notification: input language switcher. For that one, runtiem ID is undefined (empty). Given that language switch can happen everywhere, I think we should revisit the decision to limit notification handling to focused app (I think our justification back then (annoyance) does not hold well); that's for another issue.

Thanks.

I would be uncomfortable accepting this pr as there is no clear evidence that the second value is always a native window handle. Clearly there is something special going on between narrator and VoiceAccess. This really needs to be talked about with Microsoft, specifically the VoiceAccess team. I am happy to reach out to them through my other Microsoft contacts and get them connected with you @josephsl if that would help?

Hi,

I came to the same conclusion. At this point, there is no direction, and turns out the PR won't work with portable NVDA anyway. It could be a good use case for add-ons but not up to par with NVDA Core code quality. I think we could also raise a point about File Explorer/shell/input switch notification event as this is perhaps the weirdest element we came across (no runtime ID at all).

Thanks.

Closing based on above comments. Hopefully it can be addressed in VoiceAccess itself.