NVDAObjects.UIA: announce drag and drop target effects#14101
Merged
Conversation
…ss#12271. Revisiting NVDA issue 12271: UIA elements can provide drag drop effect, a text describing where the dragged object is located in relation to other elements. A somewhat related property, called drop target drop target effect property, is used by elements supporting drop target events to describe what happened when elements were dragged and dropped (rearranged in some cases). Most notably, the former property changes when rearranging Windows 10 Start menu tiles, and the latter property changes when rearranging virtual desktops in Windows 11. As a first step, recognize drag drop effect property changes, with the actual implementation coming in the next commit.
…#12271. The other side of drag and drop is drop target events set and their effects on the item being dragged and/or drop target element. For drop target effect, it is the element to which the dragged item can be dropped into, meaning it affects a different element other than the item being dragged at the moment. Therefore let NVDA anounce drop target effect property changes if given a chance to do so (implementation is the next commit).
…ess#12271. Handle drop target effect property changes, hopefully from the object that actually is the target of a drag and drop operation. However there are cases where it is not the actual drop target element that records the effect property, notably when rearranging quick settings items in Windows 11. Therefore traverse focus ancestors until an element with drop target effect property is found and then report it as the effect property. The implication of this corner case is that drop target property change must be fetched globally (next commit).
…erty change event. Re nvaccess#12271. Because drop target effect property can come from an element that is not necessarily the immediate focus element, treat it as though it is a global event. This is more so in Windows 11 Version 22H2 where selective UIA event registration is the default and drop target effect changes when rearranging quick settings items in Windows 11.
Contributor
Author
|
CC @Mazen428 |
See test results for failed build of commit 3792e90afe |
…access#12271. Because a drop target element can tell UIA that drop target effect can change many times, it becomes hard to notify users about latest happenings iwth drag and drop operation. Therefore cancel speech every time drop target effect property changes, similar to layout invalidated event where suggestions count changes multiple times.
…s. Re nvaccess#12271." This reverts commit cd8f1d0 as it causes 'dragging' to not be announced when using the keyboard to rearrange tiles in Windows 10. It still causes effects to be announced continuously when using a mouse, so a different approach for dealing with input sources other than keyboards may need to be investigated.
See test results for failed build of commit 655a877d09 |
seanbudd
reviewed
Sep 5, 2022
seanbudd
left a comment
Member
There was a problem hiding this comment.
Could system tests be added using the manual testing example?
| globalEventHandlerGroupUIAPropertyIds = { | ||
| UIA.UIA_RangeValueValuePropertyId | ||
| UIA.UIA_RangeValueValuePropertyId, | ||
| UIA.UIA_DropTargetDropTargetEffectPropertyId |
Member
There was a problem hiding this comment.
added comma for easier diffs
Suggested change
| UIA.UIA_DropTargetDropTargetEffectPropertyId | |
| UIA.UIA_DropTargetDropTargetEffectPropertyId, |
6 tasks
Contributor
Author
|
Hi, we can try coming up with system tests – note that rearranging Start menu tiles could work best as Windows Server 2019 (the one Appveyor is running on) qualifies as a Windows 10 system i.e. Windows 10 Version 1809 (will need to invest in some time learning about Robot framework). In the meantime I’ll apply the comma change. Thanks.
|
Member
|
@josephsl - yes I think the system test would need to only run on the known versions of Windows with the behaviour of the test. |
Contributor
Author
|
Hi, right – that’s what I’m afraid of, as I remember a discussion about adding Windows specific tests a while back. Thanks.
|
Contributor
Author
|
Hi, I think system tests cannot be done easily for this PR as drag and drop target effect text begins with a specific keyword ("place"). I'm sure it might be possible to detect substrings (text in str) but may need to read Robot framework docs more thoroughly. I think we might as well come up with an issue/PR pair (if it doesn't exist yet) just to deal with foundations to support Windows release detection (perhaps using sys.getwindowsversion function) and ways to detect substrings, as it can make future iterations of PR's like this easier to test locally in a more systematic way. For anyone wishing to tackle system tests for PR's like this, you must first detect which Windows release the test system is running (perhaps using sys.getwindowsversion().platform_version), then emulate key presses (see existing system test robot files for details) and figure out what NVDA is saying or get a speech substring somehow. Thanks. |
seanbudd
approved these changes
Sep 5, 2022
LeonarddeR
reviewed
Sep 5, 2022
…is is not a UIA object Suggested by Leonard de Ruijter: just continue the ancestor retrieval loop if NVDA came across a non-UIA object as it avoids handling COM error. Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com>
Suggested by Leonard de Ruijter: now that only COM error is handled, there is no need to handle attribute error as cacheable value will not be fetched for non-UIA objects. Co-authored-by: Leonard de Ruijter <leonardder@users.noreply.github.com>
See test results for failed build of commit a66e8da9b2 |
seanbudd
approved these changes
Sep 6, 2022
Contributor
Author
|
Hi, yes, I’m using isinstance call in the add-on version – will investigate calling isinstance on self in the next round (today). Thanks for the reminder.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi,
This is more of a second attempt at #12271
Link to issue number:
Fixes #12271
Follow-up and a different approach to #12428
Summary of the issue:
When drag and drop target events happen, NVDA does not announce their effects such as screen content changes such as effect of rearranging Start menu tiles.
Description of user facing changes
NVDA will announce effects of drag and drop events in places such as:
Description of development approach
While this PR may look similar to #12428, the internals are different:
Testing strategy:
Manual and add-on based testing:
Testing drag drop effect (prerequisite: Windows 10):
Before the PR: NVDA says nothing about drag drop effect.
After the PR: NVDA will announce the new position of the dragged tile.
Testing drop target effect (for best results, use Windows 11):
Before the PR: NVDA does not say anything about virtual desktop position.
After the PR: NVDA announces effect of reordering virtual desktop such as new item position.
Known issues with pull request:
This PR takes care of drag and drop target effect texts announcement, but does not include additional work for handling drag start/cancel/complete and drop target enter/leave/dropped events themselves (at least drag start event and the resulting dragging state change is handled by #14097). Should the need arise to handle actual drag and drop target events set, foundation is there to support them.
Change log entries:
Similar to #14097 but could be expanded to say (new features or bug fixes):
NVDA will announce effects of dragging items on screen in places such as rearranging Windows 10 Start menu tiles and virtual desktops in Windows 11. (#12271)
Code Review Checklist: