NVDA objects/UIA: introduce UIA drag and drop event handlers#12428
NVDA objects/UIA: introduce UIA drag and drop event handlers#12428josephsl wants to merge 10 commits into
Conversation
…. Re nvaccess#12271. UIA drag complete and drop target dropped events are used to announce new location for an element. The former is seen when rearranging tiles in Windows 8 start screen, and the latter is triggered when arranging quick actions in Windows 10 Action Center. Although both events might as well call gain focus event handler, specific events will be defined in UIA objects for future-proofing.
When tiles are rearranged in Windows 8 start screen, braille reports new location whereas speech doesn't. Therefore use UIA drag complete event to force focus to be reported.
. Unlike UIA drag complete event, drop target dropped event is raised by a containing element where items are being rearragned, the best example being rearranging quick actions in Windows 10 Action Center. At least NVDA does know what the focused element is, therefore raise a gain focus event on the focused object so the newly dragged item can be announced.
See test results for failed build of commit 5033c9e9ac |
…ions with Flake8 F405 NOQA (definition). Re nvaccess#12271
That has happened. |
LeonarddeR
left a comment
There was a problem hiding this comment.
To be honest, I think we should put some more thought into how drag and drop should behave in a screen reader before come up with the actual implementation.
| #UIA_ToolTipClosedEventId:"hide", | ||
| UIA_Window_WindowOpenedEventId:"UIA_window_windowOpen", | ||
| UIA_SystemAlertEventId:"UIA_systemAlert", | ||
| UIA_Drag_DragCompleteEventId: "UIA_dragComplete", # NOQA: F405 |
There was a problem hiding this comment.
Please avoid noqa
| UIA_Drag_DragCompleteEventId: "UIA_dragComplete", # NOQA: F405 | |
| UIA.UIA_Drag_DragCompleteEventId: "UIA_dragComplete", |
There was a problem hiding this comment.
NOQA is required because without that, local lint test fails- Flake8 doesn't know where these UIA event constants are coming from since we import everything from UIA client support library (COM interfaces).
There was a problem hiding this comment.
Note that in my suggestion, I suggested to use the UIA import. That should avoid the Flake8 error.
| A prominent example is rearranging Start screen tiles on Windows 8 and later. | ||
| """ | ||
| # #12271: speech does not report new location but braille does. | ||
| self.reportFocus() |
There was a problem hiding this comment.
This puzzles me.
- Why are you not mimicking gainFocus behavior completely here? Especially for vision, it makes sense to communicate the new location so the highlighter will follow.
There was a problem hiding this comment.
Oops - looks like a mismatch between this PR and WinTenApps where the latter doees queue a gain focus event. I'll correct this today.
| """ | ||
| # #12271: unlike drag complete event, it is a different element that raises this event. | ||
| # but NVDA does record the newly dragged element as focused object. | ||
| api.getFocusObject().reportFocus() |
There was a problem hiding this comment.
What if the dragged element is entirely unrelated to the focused object?
There was a problem hiding this comment.
That's a distinct possibility. We may need to find more places where UIA drag/drop events are fired - so far, prominent places I found are ones I noted in code comments.
|
Note that there's also the locationChange event. Would it be possible to use that instead? |
|
hi, Location change event is defined in IAccessible/internal WinEvent handler, therefore the approach in this PR to handle UIA events. I think we might as well create another PR to let base NVDA objects handle location changes before applying it to various accessibility API handlers. I'm looking into the possibility that UIA might have an equivalent event to location change MSAA event. Thanks, |
…vaccess#12271. Pointed out by Leonard de Ruijter: obj.reportFocus does not allow vision enhancmenet providers and other components from reacting to gain focus event. Therefore call obj.event_gainFocus when drag complete and/or drop target dropped events are fired. Also, add a source code comment on drop target dropped event that points out which element fires this event.
|
Hi, I’ll look into it this weekend. Thanks.
|
…s (integers). Re nvaccess#12271. Reminder from Leonard de Ruijter: reference UIA module when defining UIA event constants (UIA.constant).
|
|
||
| def event_UIA_droptTargetDropped(self): | ||
| """ | ||
| Raised by a containing element such as a list when an element is dragged over another element. |
There was a problem hiding this comment.
| Raised by a containing element such as a list when an element is dragged over another element. | |
| Raised by a containing element such as a list when an element is dragged and dropped into another element. |
This fires on drop not drag over, right?
| Raised by a containing element such as a list when an element is dragged over another element. | ||
| A prominent example is rearranging quick actions in Windows 10 Action Center. | ||
| """ | ||
| # #12271: unlike drag complete event, it is a different element that raises this event. |
There was a problem hiding this comment.
Will both of these events ever be fired at the same time and result in double speaking?
LeonarddeR
left a comment
There was a problem hiding this comment.
I'm still a bit reluctant to give this a go.
- I don't think it is very good practice to use event_gainFocus in cases where it isn't triggered by a real gainFocus event that's coming through NVDA"s event handler.
- I still tink it would be best to take this back to the design table to come up with a description of expected user experience.
| Raised by an element when a drag operation on it is completed. | ||
| A prominent example is rearranging Start screen tiles on Windows 8 and later. | ||
| """ | ||
| # #12271: speech does not report new location but braille does. |
There was a problem hiding this comment.
i do not understand this comment.
There was a problem hiding this comment.
When I first implemented it (back in November 2019 under Windows 10 App Essentials add-on), I noticed that speech didn't report the newly dragged element but braille did. We may need to find out why and come up with replacements.
| """ | ||
| # #12271: unlike drag complete event, it is a different element that raises this event. | ||
| # Specifically, it is the containing element that raises this event. | ||
| # but NVDA does record the newly dragged element as focused object. |
There was a problem hiding this comment.
How is the newly dragged element focused by NVDA?
There was a problem hiding this comment.
I think the following traceback may explain this a bit (debug log with MSAA and UIA flags turned on, along with Windows 10 App Essentials enabled; scenario: an item in Action Center was rearranged):
IO - inputCore.InputManager.executeGesture (01:23:37.906) - winInputHook (1140):
Input: kb(desktop):alt+shift+leftArrow
DEBUG - IAccessibleHandler.internalWinEventHandler.winEventCallback (01:23:37.927) - MainThread (7948):
Hook received winEvent: EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 176, childID 0, process 2844 (shellexperiencehost), thread 10584
DEBUG - IAccessibleHandler.internalWinEventHandler.winEventCallback (01:23:37.928) - MainThread (7948):
Adding winEvent to limiter: EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 176, childID 0, process 2844 (shellexperiencehost), thread 10584
DEBUG - IAccessibleHandler.internalWinEventHandler.winEventCallback (01:23:37.941) - MainThread (7948):
Hook received winEvent: EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 177, childID 0, process 2844 (shellexperiencehost), thread 10584
DEBUG - IAccessibleHandler.internalWinEventHandler.winEventCallback (01:23:37.941) - MainThread (7948):
Adding winEvent to limiter: EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 177, childID 0, process 2844 (shellexperiencehost), thread 10584
DEBUG - IAccessibleHandler.orderedWinEventLimiter.OrderedWinEventLimiter.flushEvents (01:23:37.946) - MainThread (7948):
Emitting winEvent EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 176, childID 0, process 2844 (shellexperiencehost), thread 10584
DEBUG - IAccessibleHandler.orderedWinEventLimiter.OrderedWinEventLimiter.flushEvents (01:23:37.946) - MainThread (7948):
Emitting winEvent EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 177, childID 0, process 2844 (shellexperiencehost), thread 10584
DEBUG - IAccessibleHandler.processFocusWinEvent (01:23:37.949) - MainThread (7948):
Processing focus winEvent: window 3281400 (Windows.UI.Core.CoreWindow), objectID 177, childID 0, process 2844 (shellexperiencehost), force False
DEBUG - IAccessibleHandler.winEventToNVDAEvent (01:23:37.949) - MainThread (7948):
Creating NVDA event from winEvent: EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 177, childID 0, process 2844 (shellexperiencehost), use cache False
DEBUG - IAccessibleHandler.winEventToNVDAEvent (01:23:37.950) - MainThread (7948):
winEvent mapped to NVDA event: gainFocus
DEBUG - IAccessibleHandler.winEventToNVDAEvent (01:23:37.950) - MainThread (7948):
Native UIA window. Dropping winEvent EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 177, childID 0, process 2844 (shellexperiencehost)
DEBUG - IAccessibleHandler.processFocusWinEvent (01:23:37.951) - MainThread (7948):
Processing focus winEvent: window 3281400 (Windows.UI.Core.CoreWindow), objectID 176, childID 0, process 2844 (shellexperiencehost), force False
DEBUG - IAccessibleHandler.winEventToNVDAEvent (01:23:37.951) - MainThread (7948):
Creating NVDA event from winEvent: EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 176, childID 0, process 2844 (shellexperiencehost), use cache False
DEBUG - IAccessibleHandler.winEventToNVDAEvent (01:23:37.951) - MainThread (7948):
winEvent mapped to NVDA event: gainFocus
DEBUG - IAccessibleHandler.winEventToNVDAEvent (01:23:37.951) - MainThread (7948):
Native UIA window. Dropping winEvent EVENT_OBJECT_FOCUS, window 3281400 (Windows.UI.Core.CoreWindow), objectID 176, childID 0, process 2844 (shellexperiencehost)
DEBUG - _UIAHandler.UIAHandler.IUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent (01:23:37.982) - Dummy-3 (12344):
HandleFocusChangedEvent: Ignoring because no object or ignored by object itself
DEBUGWARNING - _UIAHandler.UIAHandler.IUIAutomationFocusChangedEventHandler_HandleFocusChangedEvent (01:23:37.985) - Dummy-3 (12344):
HandleFocusChangedEvent: Ignoring duplicate focus event
DEBUG - external:globalPlugins.wintenObjs.GlobalPlugin.uiaDebugLogging (01:23:38.021) - MainThread (7948):
W10: UIA object: <NVDAObjects.UIA.UIA object at 0x072874D0>, name: , event: dropTargetDropped, app module: <'shellexperiencehost' (appName 'shellexperiencehost', process ID 2844) at address 6cb1df0>, Automation Id: ScrollWrapper, class name: ScrollViewer
NOte that this was with selective UIA event registration enabled.
Thanks.
|
Hi, I think both events will not be raised at the same time (I need to find an example that does this, but so far I can’t find it), and drop target dropped is fired when an element is dropped (will take a look at comment suggestion while looking into Leonard’s comments). Thanks.
|
|
Hi, The resulting experience is that NVDA would announce the newly dragged element, similar in experience to how Narrator does it. Thanks. |
|
Hi, I'm willing to revisit this PR throughout this summer, but since Leonard asked that we go back to the drawing board, I think it would be best to close it for now unless someone wants to work on it further based on what we have so far. Thanks. |
|
Hi @josephsl, I think it's a good idea to leave this open for now and label as needing a new author. This PR (and others) could be picked up by NV Access if another contributor doesn't get to it first. |
|
Hi, or perhaps let’s label it as “good for new dev” for people wanting to learn more about internals of UI Automation. Thanks.
|
|
I'm happy to take this one as soon as we agree about UX.
|
|
Hi, I guess I can say it now: when in doubt, learn from what Narrator does and does not do. By the way, according to UIA docs, drag and drop target events are available from Windows 8. Thanks.
|
|
Hi, by the way, whichever UX decision is chosen, I’ll apply it to WinTenApps add-on. Thanks.
|
|
This is interesting and desirable work @josephsl, I for one would hate to
see it get closed.
Although might this be a bit much for a new dev, especially given the high
level disagreement about direction? Just a thought.
|
|
Hello, In my opinion if at least NVDA could announce that an item is draggable would be great like narrator, right now you'd have to press NVDA F 1 and look at the bottom of the log to know which isn't really the best. |
|
I don’t want to see this PR being abandoned. Does it have the latest news? |
|
@cary-rowen No it does not as it has been clearly labeled as closed/needs-new-author which means that unless someone has time and motivation to continue this work it is stalled. @seanbudd Perhaps PR's labeled as 'closed/needs-new-author' should be closed or at a minimum marked as drafts to avoid confusion? |
|
Hi, specific reason: life priorities and PR design. In the meantime Windows App Essentials add-on provides this functionality, something that is really an alternative at this point (I hope someone can make this better soon).
|
|
@lukaszgo1 - agreed, closing this PR so it can be taken over. |
|
2022 update: see josephsl/wintenApps#75 as to why this PR should be taken care of sooner than later, as without this, users must rely on Windows App Essentials add-on to rearrange tiles and folders in Windows 11 Version 22H2 start menu. |
Fixes #12271 Follow-up and a different approach to #12428 Summary of the issue: When drag and drop target events happen, NVDA does not announce their effects such as screen content changes such as effect of rearranging Start menu tiles. Description of user facing changes NVDA will announce effects of drag and drop events in places such as: - Windows 10 Start menu: after moving tiles - Windows 10 Action center/Windows 11 quick setings: afterreordering settings items - Windows 11 task view: rearranging virtual desktops - Windows 11 Start menu: rearranging pinned items, and from Version 22H2 onwards, creating app folders Description of development approach While this PR may look similar to #12428, the internals are different: - In UIA handler, added property events for drag drop and drop target effects. - In UIA objects, add property events for drag drop and drop target effects and announce effect text. - For drop target effect, because drop target event may come from somewhere other than the object that "raises" it, traverse focus ancestors looking for UIA elements with drop target effect text set. - Back in UIA handler, because drop target effect event can come from somewhere other than the focused element or its parent, treat it as a global event. --- * UIA handler: introduce UIA drag drop effect property event. Re #12271. Revisiting NVDA issue 12271: UIA elements can provide drag drop effect, a text describing where the dragged object is located in relation to other elements. A somewhat related property, called drop target drop target effect property, is used by elements supporting drop target events to describe what happened when elements were dragged and dropped (rearranged in some cases). Most notably, the former property changes when rearranging Windows 10 Start menu tiles, and the latter property changes when rearranging virtual desktops in Windows 11. As a first step, recognize drag drop effect property changes, with the actual implementation coming in the next commit. * NVDAObjects.UIA: handle UIA drag drop effect property. Re #12271. Announce drag drop effect if it can be fetched (introduced in Windows 8). * UIA handler: introduce drop target effect property event. Re #12271. The other side of drag and drop is drop target events set and their effects on the item being dragged and/or drop target element. For drop target effect, it is the element to which the dragged item can be dropped into, meaning it affects a different element other than the item being dragged at the moment. Therefore let NVDA anounce drop target effect property changes if given a chance to do so (implementation is the next commit). * NVDAObjects.UIA: handle drop target effect property changes. Re #12271. Handle drop target effect property changes, hopefully from the object that actually is the target of a drag and drop operation. However there are cases where it is not the actual drop target element that records the effect property, notably when rearranging quick settings items in Windows 11. Therefore traverse focus ancestors until an element with drop target effect property is found and then report it as the effect property. The implication of this corner case is that drop target property change must be fetched globally (next commit). * UIA handler: treat drop target effect property event as a global property change event. Re #12271. Because drop target effect property can come from an element that is not necessarily the immediate focus element, treat it as though it is a global event. This is more so in Windows 11 Version 22H2 where selective UIA event registration is the default and drop target effect changes when rearranging quick settings items in Windows 11. * UIA handler/property names to NVDA events: lint * NVDAObjects.UIA: cancel speech when drop target effect changes. Re #12271. Because a drop target element can tell UIA that drop target effect can change many times, it becomes hard to notify users about latest happenings iwth drag and drop operation. Therefore cancel speech every time drop target effect property changes, similar to layout invalidated event where suggestions count changes multiple times. * Revert "NVDAObjects.UIA: cancel speech when drop target effect changes. Re #12271." This reverts commit cd8f1d0 as it causes 'dragging' to not be announced when using the keyboard to rearrange tiles in Windows 10. It still causes effects to be announced continuously when using a mouse, so a different approach for dealing with input sources other than keyboards may need to be investigated. * UIA handler: add comma for ease of diff comparisons * update changes * NVDAObjects.UIA (optimization): do not fetch UIA property cache if this is not a UIA object Suggested by Leonard de Ruijter: just continue the ancestor retrieval loop if NVDA came across a non-UIA object as it avoids handling COM error. Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com> * NVDAObjects.UIA: remove unneeded attribute error check. Suggested by Leonard de Ruijter: now that only COM error is handled, there is no need to handle attribute error as cacheable value will not be fetched for non-UIA objects. Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com> * NVDAObjects.UIA: NVDAObjects.UIA.UIA -> UIA when checking instance. Co-authored-by: Sean Budd <sean@nvaccess.org> Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com>
Hi,
To be marked ready for review once NVDA 2021.1 beta 1 is released and master branch moves onto 2021.2:
Link to issue number:
Closes #12271
Summary of the issue:
UIA drag and drop events are used to communicate that items were rearranged on screen or dragged around. This is used when:
Description of how this pull request fixes the issue:
Introduce UIA drag complete and drop target dropped event handlers so NVDA can announce at least the dragged item and its new positoin. The event handler can be expanded to add support for vision enhancement providers and other goddies upon request.
Testing strategy:
Tested through Windows 10 App Essentials add-on:
Known issues with pull request:
None
Change log entries:
New features:
NVDA can announce new location of items such as Start menu tiles and Action Center items after items are rearranged. (#12271)
Possible changes for developers:
Added event handlers for UIA drag complete and drop target dropped events. (#12271)
Code Review Checklist:
Thanks.