Windows 11 shell (appModules/explorer): classify certain windows as UIA to allow mouse and touch navigation to work while interacting with shell elements#13691
Merged
Conversation
… windows to allow mouse and ouch navigation to work. Re nvaccess#13506. When using the mouse and/or touch to navigate Windows 11 shell elements such as tskbar and other buttons, NVDA says 'DesktopWindowXamlSource'. This window title is employed from desktop (Win32) apps hosting XAML controls. Reclasifying these controls as well as some top-level windows from IAccessible to UIA does allow mouse and touch interaction to work. Because DesktopWindowXamlSource class label is seen in apps such as Notepad and Windows Terminal (in addition to File Explorer), a comment was added to indicate that a more global solution would be better. But for now at least shell elements can be navigated with mosue and/or touch.
…ne when checking Windows 11 shell elements. Re nvaccess#13506. Turns out checking just the class name is not enough for shell root window (Shell_TrayWnd) - one needs to check ancestors. Because other windows are top-level windows (such as Task View), class name check will work, but for consistency, call winUser.getAncestor with root (winUser.GA_ROOT (2)) flag. Also add an explanation as to why.
…s release. Re nvaccess#13506. Small optimization: winVersion.getWinVer() can be called up to two times (if running on Windows 11), therefore use a variable to store current Windows release and use it throughout the method.
See test results for failed build of commit 07c73dc042 |
Member
|
Which aspect of #13506 will still be an issue after this PR? |
Contributor
Author
|
Hi, the inability to use mouse and/or touch to navigate menu bar in File Explorer and Notepad mentioned in #13506 cannot be resolved until a more general solution to deal with DesktopWindowXamlSource is found across apps. Thanks.
|
Contributor
Author
|
Hi, As for a new issue on DesktopWindowXamlSource, general issue should be created to resolve this at a more global level, but the chnace of a pull request from me is low (at the moment). Thanks. |
seanbudd
approved these changes
May 19, 2022
5 tasks
seanbudd
pushed a commit
that referenced
this pull request
May 23, 2022
…IA element (#13723) Fixes a possible regression from #13691 Follow-up to #13506 Fixes #13717 Summary of the issue: On some Windows 11 systems, Start button is seen as IAccessible object rather than UIA element. This causes IAccessible handler to report attribute error when handling events as an attribute is not present in UIA elements. Description of how this pull request fixes the issue: On the surface, Start button window class name (Start) should be classified as a bad UIA window. However, when handling events from UIA handler thread, good UIA windows are checked before looking for bad UIA window class names. Because File Explorer's good UIA window method checks for the root element (the element being the ancestor of Start button), bad UIA window class name will not et a chance to detect IAccessible Start button. Therefore edit good UIA window method instead by checking for the specific class name in Windows 11. Testing strategy: Manual testing: Prerequisites: Windows 11 Version 21H2 or later (test systems are running Version 22H2 preview (build 22621 beta): Run a build from this branch. Press Windows+D to show Desktop. Press Tab to move through shell elements such as Start button, Task View button, Taskbar, notification area, and back to Desktop. While performing step 3, make sure NVDA is saying something. For detailed debug output, enable IAccessible and UIA debugging from Advanced settings panel, restart NVDA with debug logging enabled, then perform steps 2 through 4 from aboveabove. Then open the log and see what IAccessible and UIA handlers are saying about focus events. * appModules/explorer: do not treat IAccessible Start button class as UIA window. Re #13717. Reported by a user: on some systems, Windows 11 Start button is seen as IAccessible (MSAA), not UIA, causing IAccessible handler to report AttributeError since event ID could not be found in UIA objects. Therefore in File Explorer app module, do not treat IAccessible Start button (window class name: Start) as UIA element. * appModules/explorer: combine Start button IAccessible object class check with other class name checks. Re #13717. Tip from Sean Budd (NV Access): move Start button class name check to the parent boolean - that is, package Windows 11 + shel elements + excluding IAccessible Start button under 'logical and' check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to issue number:
#13506 (partial)
Summary of the issue:
Windows 11 UI (shell) elements such as taskbar, notification area, TaskView, Snap layouts, and buttons such as Search and Chat cannot be navigated to and interacted with mosue and touchscreens.
Description of how this pull request fixes the issue:
Reclassifies the following windows as UIA windows:
Most of these have the label of "DesktopWindowXamlSource". The "isGoodUIAWindow" in File Explorer app module was edited to treat these windows as UIA elements provided that they are the ancestors (top-level windows) of windows passed into that method. This is more so for the first window class name (shell root) whereas others can go through just a class name check, but cal winUser.getAncestor with GA_ROOT (2) flag for all windows for consistency.
Testing strategy:
Manual testing (system tests cannot be added until Appveyor uses a server version based on Windows 11):
To check that mosue and touch navigation works, make sure you don't hear "DesktopWindowXamlSource" for the scenarios described (there might be other windows with this label but are out of scope of this PR).
Known issues with pull request:
This solution is specific to Windows 11 shell. There are desktop apps that embed XAML controls, hence the class name of "DesktopWindowXamlSource", and other app examples include Notepad, Windows Terminal, and This PC (file browser part of File Explorer). A general solution that works across apps is preferred.
Change log entries:
Bug fixes:
In Windows 11, it is again possible to navigate and interact with user interface elements such as Taskbar and Task View using mouse and touch interaction. (#13508
Code Review Checklist: