Add a native selection mode to Gecko virtualBuffers#15830
Conversation
…Accessible2_4 which provides SetSelectionRanges, already implemented in google Chrome.
… virtualBuffer xml content so that it is possible to know how frar into a node the requested offsets actually cover. Assists in converting a virtaulBuffer offset to an IAccessible2 text offset.
…rmatFields to assist in looking up an IAccessible2 object + text offset from a given formatField, needed for performing DOM selections.
…om the current browse mode selection. This makes use of a new updateAppSelection method on browseMode documents. Implement updateAppSelection for Chromium virtualBuffers (which uses IAccessible2_4::SetSelectionRanges) and UIA which uses IUIAutomationTextRange::Select.
…unction as this is now in gecko_ia2's vbuf which is a super class.
…windowHandle on controlFields that represent embedded objects within an IAccessibleText run. Gecko_ia2 virtualBuffer: support looking for IAccessible2 text start offset, uniqueID and windowHandle attributes on controlFields, not just formatChanges, so that we can select embedded objects, where the vbufBackend has not rendered the specific IAccessibleText run within the embedded object as it had an overriding name or did not support IAccessibleText (E.g. a graphic).
… toggled on and off with NVDA+shift+f10. When on, the app selection mirrors the virtualBuffer selection, and copy (control+c) is handled entirely by the app.
…and UIA for now. Just leaving it in the Gecko_ia2 virtualBuffer. It can be abstracted more broadly once we have an acceptable implementation in one browser.
… controlEnd commands.
See test results for failed build of commit dd3480c493 |
|
Because this operation encountered a DLL that is being referenced by another process, see #15884. The fastest way is to log out the current user of Windows |
|
After restarting the computer, the procedure suggested by @lukaszgo1 worked to build NVDA, and now the native selection mode works as expected. |
Yes, this PR is already more than useful in the current state. It should not be blocked by other questions (including mine), |
|
@hwf1324 wrote:
So you can read the mouse selection out by pressing nvda+shift+up arrow in desktop or nvda+shift+s in laptop keyboard layout both while you are in browse mode and in focus mode? |
| ) | ||
| def script_copyToClipboard(self, gesture): | ||
| if self._nativeAppSelectionMode: | ||
| ui.message(_("native copy")) |
There was a problem hiding this comment.
@michaelDCurran - thoughts on naming this?
Ideas:
- application copy
- rich copy
- formatted copy
|
Thinking of the messages to the user, "Copy with formatting" is probably about as intuitive as I can think of. |
Unfortunately, in this PR situation: in browsing mode, it reports "no text selected"; in focus mode, it reports "text selected". |
The legacy copy message reportes "Copied to clipboard" followed by either the copied text, or the number of characters of this text if it is high enough. I do not see any reason to have a different message for formatted copy. |
|
If there's negative feedback with "native copy" we can consider changing it in future, or removing the message entirely |
|
Actually I find the message quite useful, I know that the setting is enabled at least. Note that the native selection turns off automatically when you switch to a new tab or a new website in Firefox. Is this intended behavior? |
Summary of the issue: NVDA allows the user to select text in browse mode. In web browsers where NVDA uses a virtualBuffer to make the content accessible, the selected text is only a plain text representation, and thus copying this text to another application such as MS Word looses all rich formatting. It is also impossible for the user to perform a web page specific action on the selected text, as the web page / application is unaware of NVDA's selection. E.g. Hypothesis's annotation and highlighting features. Description of user facing changes A new Native Selection mode (toggled by NVDA+shift+f10) is now available in NVDA's browse mode for Mozilla Firefox. When turned on, selecting text in browse mode will also manipulate Firefox's own native selection. And copying text with control+c will pass straight through to Firefox, thus copying the rich content, rather than NvDA's plain text representation. Description of development approach Added an updateAppSelection method to the Gecko virtualBuffer Python class, which updates the native selection to mirror NVDA's current browse mode selection, by using the new IAccessibleTextSelectionContainer interface in IAccessible2. This change also required the following: The gecko virtualBuffer backend exposes needed attributes such as IAccessible2 uniqueID on all text runs, so it can be later used to update the native selection via IAccessibleTextSelectionContainer. The base virtualBuffer storage generated XML also includes offsetFromStartOfNode and offsetFromEndOfNode on text tags, so that it is clear how far in the range starts from the real start and end of the text node. XMLTextParser in NvDA now maintains a controlField stack and controlEnd fields now include a reference to the same attribs dictionary that was used on the controlStart. This is so we have all the information about a field by only looking at its controlEnd. Such as when traversing through fields backwards. The Gecko virtualBuffer textInfo's updateSelection method calls updateAppSelection if native selection mode is toggled on.
Because it's only when NVDA (probably any screen reader) inserts itself into the copy process via using the virtual buffer that formatting can be stripped. When a screen reader is not in use copy from a webpage is always WYSIWYG, but not so when using NVDA in "regular" mode versus native text selection mode. If it happens to be easy for NVDA to alert the user whether a formatted copy versus an unformatted copy has occurred, to me that has value. |
|
Just curious if there is any active pursuit of native text selection under Chromium-based browsers that's in the works? |
|
Just wanted to mention that with Caret browsing on (F7) and NVDA manually switched to forms mode (Insert Space) native selection happens in Chrome, but it happens without NVDA speaking the selection. Perhaps this could be an avenue toward a solution for Chrome? |
|
I believe the Chromium bugs with IAccessibleTextSelectionContainer are fixed in the latest Canary, so NVDA could try turning on support again. Let us know if there are new bugs that you see! |
…that can support it. (#17838) Fixes #17591 Summary of the issue: In #15830, NVDA browse mode gained a new native Selection Mode (toggled on with `NVDA+shift+f10`) which caused the browser's underlying native selection (DOM selection) to mirror the browse mode virtual selection. Although this worked well in Firefox, it had to be forcefully disabled for Chromium-based browsers as there were major bugs in Chromiums implementation of IAccessibleTextSelectionContainer. As of Chromium 134 stable, all known bugs have been addressed, thus NVDA should allow native selection mode in Chromium-based browsers where possible. Description of user facing changes NVDA browse mode's native selection mode (`NVDA+shift+f10`) can now be enabled in any browser based on Chromium 134 or newer. Description of development approach * The Chromium virtualBuffer is no longer marked as not supporting native app selection mode. * toggling native selection mode on now firstly tries to clear the native selection before updating it, and also catches more errors, ensuring that native selection mode is truly supported when enabling. Testing strategy: * In Firefox, visited nvaccess.org, enabled native selection mode, selected the navbar, copied to clipboard, and pasted to an editor to verify all expected content was there. * In Chrome134, visited nvaccess.org, enabled native selection mode, selected the navbar, copied to clipboard, and pasted to an editor to verify all expected content was there. * In MS Edge 135, visited nvaccess.org, enabled native selection mode, selected the navbar, copied to clipboard, and pasted to an editor to verify all expected content was there. * In VS Code February update (Chromium 132), in browse mode tried to enable native selection mode, which reported as not supported as expected. Known issues with pull request: None known.
Link to issue number:
None
Summary of the issue:
NVDA allows the user to select text in browse mode. In web browsers where NVDA uses a virtualBuffer to make the content accessible, the selected text is only a plain text representation, and thus copying this text to another application such as MS Word looses all rich formatting. It is also impossible for the user to perform a web page specific action on the selected text, as the web page / application is unaware of NVDA's selection. E.g. Hypothesis's annotation and highlighting features.
Description of user facing changes
A new Native Selection mode (toggled by NVDA+shift+f10) is now available in NVDA's browse mode for Mozilla Firefox. When turned on, selecting text in browse mode will also manipulate Firefox's own native selection. And copying text with control+c will pass straight through to Firefox, thus copying the rich content, rather than NvDA's plain text representation.
Description of development approach
Testing strategy:
Known issues with pull request:
Chromium implemented IAccessibleTextSelectionContainer, however much of the time its offset calculation seems to be wrong, thus the wrong native selection is made, thus this cannot be supported in Chromium at this point in time.
Code Review Checklist: