Windows 11 22H2: add good UIA window class names for system tray overflow window and Open With dialog to allow mouse and touch interaction#14540
Merged
seanbudd merged 10 commits intoJan 16, 2023
Conversation
… window. Re nvaccess#14538. Windows 11 22H2 (at least newer updates) include modernized Open With dialog powered by WinUI, meaning no proper mouse and/or touch interaction. Therefore treat the top window for this dialog as a good UIA window so mouse and touch interaction can work properly. Also, rewrite isGoodUIAWindow method with one window class name per line.
…ood UIA window. Re nvaccess#14539. Later revisions of Windows 11 Version 22H2 (2022 Update) comes with redesigned system tray overflow window (class name: TopLevelWindowForOverflowXamlIsland) but without proper mouse and touch interaction from NVDA. Therefore, add the new class as a good UIA window so mouse and touch interaction can work.
See test results for failed build of commit 45dfc05319 |
Contributor
|
Hello. |
Contributor
Author
|
Hi, tried this to no avail at the moment – it does not work for touch users, although further testing is needed to determine what’s going on. Thanks.
|
Contributor
Author
|
Hi, Just telling NVDA that XAML source windows are UIA won't be enough - it is part of aps (specifically, WinUI/XAML Islands), and it is really the top-level windows that are seen as IAccessible, thus these windows must be the ones reclassified as UIA. We can't also rely on the idea that XAML source windows will have the same window class names across apps - class names are different between File Explorer and Windows Terminal. Most importantly, if XAML source window is reclassified as a UIA element, it leads to NVDA becoming unresponsive at startup for several seconds. Thanks. |
seanbudd
reviewed
Jan 16, 2023
Contributor
Author
|
Hi, ah, wasn’t sure which type to apply for window handle… Thanks.
|
Co-authored-by: Sean Budd <seanbudd123@gmail.com>
seanbudd
reviewed
Jan 16, 2023
| # #11335: Open With dialog isn't read in Windows 10 Version 2004 (May 2020 Update). | ||
| # Note that treating the below window as a UIA window will make NVDA no longer announce "pane". | ||
| if winUser.getClassName(hwnd) == "Shell_Flyout": | ||
| from winAPI.types import HWNDValT |
Member
There was a problem hiding this comment.
You should probably pull these imports out to the top of the file
…dule. Suggested by Sean Budd (NV Access): move winAPI import to the top of the modules.
seanbudd
approved these changes
Jan 16, 2023
…penWithGoodUIAWindow
|
Thanks for the update!
…On Sun, Jan 15, 2023, 6:58 PM Sean Budd ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In source/appModules/openwith.py
<#14540 (comment)>:
> @@ -40,9 +40,15 @@ def chooseNVDAObjectOverlayClasses(self, obj, clsList):
elif automationId == "ImmersiveOpenWithFlyout":
clsList.insert(0, ImmersiveOpenWithFlyout)
- def isGoodUIAWindow(self, hwnd):
- # #11335: Open With dialog isn't read in Windows 10 Version 2004 (May 2020 Update).
- # Note that treating the below window as a UIA window will make NVDA no longer announce "pane".
- if winUser.getClassName(hwnd) == "Shell_Flyout":
+ from winAPI.types import HWNDValT
You should probably pull these imports out to the top of the file
—
Reply to this email directly, view it on GitHub
<#14540 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AXT2QUIVEZQAU7VO7FWZUCDWSSTMPANCNFSM6AAAAAAT3PQB5Q>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to issue number:
Closes #14538
Closes #14539
Summary of the issue:
Mouse and/or touch interaction does not work properly for Windows 11 22H2 UI, including upcoming redesigned system tray overflow window and Open With dialog.
Description of user facing changes
Mouse and touch interaction will work when opening system tray overflow window and Open With dialog in Windows 11 2022 Update.
Description of development approach
Edited "isGoodUIAWindow" found in File Explorer and Open With app modules to classify the following window class names as good UIA windows:
Along with this change, type information were added to these methods.
Testing strategy:
Manual testing (prereq: mouse and/or touchscreen; for mouse, mouse tracking should be turned on and NVDA sould be told to follow mouse from review cursor settings panel):
Redesigned system tray overflow window (Windows 11 22H2 build 22623 and later): press Windows +B to move focus to "show hidden icons" button, press Space to expand it, then use the mouse to move around the overflow window (located above the icons button).
Open With dialog: click on a file type that Windows cannot open (such as a pickle file) and use the mosue and/or touch to navigate and interact with the dialog.
Known issues with pull request:
None
Change log entries:
Bug fixes (under Windows 11 section):
In Windows 11 22H2, it is again possible to use mouse and touch interaction to navigate and interact in areas such as redesigned system tray overflow window and Open With dialog. (#14538, #14539)
Code Review Checklist: