Skip to content

Move the system caret when routing review cursor#15064

Merged
seanbudd merged 12 commits into
nvaccess:masterfrom
LeonarddeR:brailleRouteReviewCursorAndSystemCaretAlternative
Jul 11, 2023
Merged

Move the system caret when routing review cursor#15064
seanbudd merged 12 commits into
nvaccess:masterfrom
LeonarddeR:brailleRouteReviewCursorAndSystemCaretAlternative

Conversation

@LeonarddeR

@LeonarddeR LeonarddeR commented Jun 28, 2023

Copy link
Copy Markdown
Collaborator

First of all, many thanks to @burmancomp for laying the foundation of this pr.

Link to issue number:

Fixes #14885
Fixes #3166

Summary of the issue:

It has not been possible to move system focus/caret with single routing button press when braille is tethered to review, or when braille is tethered to automatically and follows temporarily review cursor. When user wants to review editable text without affecting system caret position (braille tethered to review continuously or temporarily), it maybe useful if he/she can route caret when needed for example to correct typos. Although thiskinds of routing has been possible, it has required at least one extra routing button press.

Description of user facing changes

A new setting Move system caret when Routing review cursor is added:

  • Never (Default): No change
  • Only when tethered automatically: Routing keys will move the caret or focus when tethered automatically. When explicitly tethered to review, routing keys won't move the caret
  • Always: Regardless whether tethered to review or automatically, routing the review cursor will move the caret or focus

Description of development approach

This pr takes a different approach to #15044.

The logic is not in the routing scripts in globalCommands but instead on the braille.ReviewTextInfoRegion class as well as two new region types for review cursor and objects. First of all, TextInfoRegion.routeTo has been split into two methods. This ensures that we can override the routing behavior for the review cursor regions without touching the BrailleInput logic. Furthermore, the following changes were added/changed:

  • ReviewNVDAObjectRegion: simply sets focus to the object before executing the default action
  • ReviewCursorManagerRegion. Inherrits from ReviewTextInfoRegion and CursorManagerRegion to ensure that ReviewTextInfoRegion._routeToTextInfo calls the right super class on cursor managers
  • ReviewTextInfoRegion: implemented _routeToTextInfo to support the new behavior. It always calls the super class to execute the normal routing action (i.e. moving the review cursor and activating the position). When routing should move the system caret, it calls setFocus on an object when the underlying TextInfo class is DisplayModel, i.e. when in screen review. Otherwise, it calls setCursor on the super class, which translates to moving the system or cursor manager caret.

Testing strategy:

Tested multiple times by @burmancomp and @irrah68. Also implemented unit tests for the review textinfo part.

Known issues with pull request:

None known.

Change log entries:

New features:

Code Review Checklist:

  • Pull Request description:

    • description is up to date
    • change log entries
  • Testing:

    • Unit tests
    • System (end to end) tests
    • Manual testing
  • API is compatible with existing add-ons.

  • Documentation:

    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • UX of all users considered:

    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • Security precautions taken.

@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit dd426737d1

@burmancomp

Copy link
Copy Markdown
Contributor

@LeonarddeR focusing objects should be possible in all modes:

  • why user should change review mode if he/she wants to focus objects; he/she can navigate objects also in other modes so why he/she could not focus them
  • if review is screen there seems to have some issues (or at least it seems to work differently) showing braille when tethered to review

Could you implement gesture also so that display manufacturers/users could define key combination if they want?

I have not tested this build or reviewed code but suppose that it works as described.

@michaelDCurran michaelDCurran left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm generally happy with this over #15044 but will leave final review for code/UI to @seanbudd

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@LeonarddeR focusing objects should be possible in all modes:

I have not tested this build or reviewed code but suppose that it works as described.

Probably better test this before deciding on whether we want to do something special with focusing. Note that changing the system caret already means an implicit focus of the object in many cases.

Could you implement gesture also so that display manufacturers/users could define key combination if they want?

Yes, there is a script available for this.

@burmancomp

Copy link
Copy Markdown
Contributor

@LeonarddeR focusing objects should be possible in all modes:

I have not tested this build or reviewed code but suppose that it works as described.

Probably better test this before deciding on whether we want to do something special with focusing. Note that changing the system caret already means an implicit focus of the object in many cases.

Many situations screen review is quite useless because braille shows nothing - try for example in notepad or write comment to this discussion with browser. It means that user should first change review to screen then focus object and then change review back for example to object. Why should he/she take these extra steps if he/she anyway wants to focus objects using routing buttons. He/she can try to focus that object anyway with other gesture, so what is advantage if user has to use other gesture to perform task he/she wants to do. And as you wrote focus is set implicitly anyway in many cases. This sounds quite confusing. I think better is to try to change focus if needed and to report to user if it fails. Because review cursor - I suppose - is moved anyway user may think that also focus/caret was moved. If braille is tethered to review braille shows review cursor all the time (if cursor is shown) so user cannot conclude that focus/caret was moved as well.

I hope you would change focusing behavior.

I set review to screen and tried navigate from other application (command prompt/terminal) to notepad and activat notepad with routing button. There was error in log and focus was not in notepad:

Input: br(albatross):routing
DEBUGWARNING - displayModel.DisplayModelTextInfo.get__storyFieldsAndRects (13:14:10.387) - MainThread (12284):
AppModule does not have a binding handle
ERROR - scriptHandler.executeScript (13:14:10.387) - MainThread (12284):
error executing script: <bound method GlobalCommands.script_braille_routeTo of <globalCommands.GlobalCommands object at 0x05F8BDD0>> with gesture 'routing'
Traceback (most recent call last):
File "scriptHandler.pyc", line 295, in executeScript
File "globalCommands.pyc", line 3523, in script_braille_routeTo
File "braille.pyc", line 2327, in routeTo
File "braille.pyc", line 1748, in routeTo
File "braille.pyc", line 1427, in routeTo
File "braille.pyc", line 1363, in routeTo
File "textInfos_init
.pyc", line 643, in activate
File "baseObject.pyc", line 41, in get
File "textInfos\offsets.pyc", line 513, in _get_pointAtStart
File "displayModel.pyc", line 482, in _getBoundingRectFromOffset
LookupError

Could you implement gesture also so that display manufacturers/users could define key combination if they want?

Yes, there is a script available for this.

Fine.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@LeonarddeR focusing objects should be possible in all modes:

I have not tested this build or reviewed code but suppose that it works as described.

Probably better test this before deciding on whether we want to do something special with focusing. Note that changing the system caret already means an implicit focus of the object in many cases.

I think better is to try to change focus if needed and to report to user if it fails.

I hope you would change focusing behavior.

I'm not convinced honestly. Could you please provide some concrete reproducible steps to prove your point?

I set review to screen and tried navigate from other application (command prompt/terminal) to notepad and activat notepad with routing button. There was error in log and focus was not in notepad:

Input: br(albatross):routing DEBUGWARNING - displayModel.DisplayModelTextInfo.get__storyFieldsAndRects (13:14:10.387) - MainThread (12284): AppModule does not have a binding handle ERROR - scriptHandler.executeScript (13:14:10.387) - MainThread (12284): error executing script: <bound method GlobalCommands.script_braille_routeTo of <globalCommands.GlobalCommands object at 0x05F8BDD0>> with gesture 'routing' Traceback (most recent call last): File "scriptHandler.pyc", line 295, in executeScript File "globalCommands.pyc", line 3523, in script_braille_routeTo File "braille.pyc", line 2327, in routeTo File "braille.pyc", line 1748, in routeTo File "braille.pyc", line 1427, in routeTo File "braille.pyc", line 1363, in routeTo File "textInfos__init_.pyc", line 643, in activate File "baseObject.pyc", line 41, in get File "textInfos\offsets.pyc", line 513, in _get_pointAtStart File "displayModel.pyc", line 482, in _getBoundingRectFromOffset LookupError

That error looks pretty unrelated, it seems to have been caused by the second routing key press.

@burmancomp

Copy link
Copy Markdown
Contributor

I think better is to try to change focus if needed and to report to user if it fails.
I hope you would change focusing behavior.

I'm not convinced honestly. Could you please provide some concrete reproducible steps to prove your point?

Steps to focus object:

  • change review to screen (which you likely do not use normally); likely your display is showing empty now
  • navigate to object (with speech because braille likely shows nothing appropriate)
  • press routing button (if object is edit field you likely press wrong button because braille shows nothing)
  • change back to other review (review to screen for example)
  • correct cursor position because it was likely wrong

I set review to screen and tried navigate from other application (command prompt/terminal) to notepad and activat notepad with routing button. There was error in log and focus was not in notepad:

That error looks pretty unrelated, it seems to have been caused by the second routing key press.

This appears with first press when most left routing button is pressed. I cannot get focus moved from command prompt to notepad.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

Based on above comment, it is really not clear to me why something should change here. Screen review only works in Win32 and legacy applications and as you say, you normally wouldn't use it unless you have a particularly good reason for it. Furthermore, focusing is already implemented for screen review as screen review doesn't offer a real caret.

@burmancomp

Copy link
Copy Markdown
Contributor

I do not know what disadvantages to possibility to focus objects would have. I think it would be nice additional possibility.

Compared with #15044 routing buttons work differently in browse mode. In #15044 routing buttons work as in braille would be tethered to focus although braille is tethered to review. This is handy when for example user wants to select and copy some text to clipboard with standard select and copy. As user I would not like to change tether to or use review mark and copy commands in thiskind of situation. I suppose #15044 works like Jaws in this situation.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I do not know what disadvantages to possibility to focus objects would have. I think it would be nice additional possibility.

Well, I'm viewing it from the other side, if there is no clear advantage in focusing, I see no reason why we should focus.

Compared with #15044 routing buttons work differently in browse mode. In #15044 routing buttons work as in braille would be tethered to focus although braille is tethered to review. This is handy when for example user wants to select and copy some text to clipboard with standard select and copy. As user I would not like to change tether to or use review mark and copy commands in thiskind of situation. I suppose #15044 works like Jaws in this situation.

Could you please elaborate? What does #15044 do in browse mode this pr doesn't?

@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit d84699e88a

@burmancomp

Copy link
Copy Markdown
Contributor

I do not know what disadvantages to possibility to focus objects would have. I think it would be nice additional possibility.

Well, I'm viewing it from the other side, if there is no clear advantage in focusing, I see no reason why we should focus.

Advantage is that if you are doing object navigation then you can simple press routing button on object to focus it - no additional steps needed. Albatross has object navigation gestures so it would be fine if one could set focus with routing button, and as said implicitly focus is set many times.

Compared with #15044 routing buttons work differently in browse mode. In #15044 routing buttons work as in braille would be tethered to focus although braille is tethered to review. This is handy when for example user wants to select and copy some text to clipboard with standard select and copy. As user I would not like to change tether to or use review mark and copy commands in thiskind of situation. I suppose #15044 works like Jaws in this situation.

Could you please elaborate? What does #15044 do in browse mode this pr doesn't?

  • tether braille to review
  • set Move system caret when Routing review cursor to always
  • open some page in browser
  • make sure browse mode is active
  • scroll with your display right/left/up/down keys somewhere from your current location
  • press NVDA+L and then press shift+NVDA+. different lines should be reported
  • then press routing button
  • give above line reporting commands; with route review cursor and system caret #15044 both gestures now reports same line what you can also read from braille line at the moment

Consider you pressed routing button because you want start text selection from that point. With #15044 for example shift+right arrow selects character where you pressed routing button, shift+end would select current line to the end etc. This implementation starts selection from the position you were before scrolling with left/right/up/down. If you however are about to select text from the position where you are now (after scrollng with left/right/up/down) you should enter at least one extra gesture. Likely user has used to think that in browse mode after routing button press text selection with standard selection commands (shift+something) is possible.

Although #15044 affects on routing button press results in browse mode when braille is tethered to review (and route review cursor along with system caret is enabled) I suppose most of users expect that in browse mode and they do not want change tethering to get "usual behavior" when using browse mode. I suppose that user who wants move system caret when routing review cursor do not want lose current browse mode behavior and that he/she does not want to change tether back and forth. I do not know if profiles were working solution. How many profiles user would need to cover all applications where browse mode is needed, and what if same application have editor (in word one can use both browse and focus mode for example).

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

This is indeed a real issue I need to investigate. The problem is actually quite simple, browse mode uses a CursorManagerRegion whereas the review cursor uses a ReviewTextInfoRegion. Both inherit from TextInfoRegion but the cursor manager logic in CursorManagerRegion is actually pretty important.

@irrah68

irrah68 commented Jun 29, 2023

Copy link
Copy Markdown

I'm writing this from an ordinary user's point of view.

I downloaded and installed "nvda_snapshot_pr15064-28517,d84699e8.exe". Braille is tethered to Review and Move system caret when routing Review cursor is set to Always.

When I navigate to an object using my Braillex EL 80c Braille display, I noticed the same problem @burmancomp has mentioned. I went to command prompt, started object navigation and found (for Example) OmniPage 18. I pressed a touc cursor key on letter O but nothing happened. I then pressed once shift + NVDA + Numpad Minus. After that I again pressed the TC key on letter O (OmniPage). Now OmniPage started. I needed an extra step to get OmniPage started. It would be easier and more practical if I could have started the program at the single TC key press. I think that if an user could do different task by using object navigation and use a single key press of a touch cursor key to get things done would be a significant improvement to NVDA and make things easyer, faster and more efficient.

I also tested copying text from Web pages. When I read a couple of lines text, pressed a touc kursor key and then pressed Shift + End in orde to get the line (that I just read from my Braille display) selected – so I could copy it to clipboard – it copied the line from the point where I was before I started reading the text. I think it should be possible to use the Review cursor and Touc Cursor keys to mark the position, from which an user can copy text to clipboard by one way or another (Shift + End, Ctrl + Shift + End etc).

It doesn't sound a good idea that when an user reads lines after lines using her/his Braille display, she/he would have to go back to the position from which she/he began to read, when the original purpose is to copy the last line of text that she/he just read using her/his Braille display. "Just press a key and copy" sound better.

@burmancomp

Copy link
Copy Markdown
Contributor

As to focusing objects with routing buttons always when it is possible, in addition to @irrah68 wrote, @LeonarddeR what do you think if it would help blind deaf users. Because they are more dependent on braille output their hands are more often on braille line. Could it have positive effect on their user experience when there would be less need to move hands back and forth between keyboard and braille line?

I am sure that next version will not suffer from browse mode problem, thank you.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author
* tether braille to review

* set Move system caret when Routing review cursor to always

* open some page in browser

* make sure browse mode is active

* scroll with your display right/left/up/down keys somewhere from your current location

* press NVDA+L and then press shift+NVDA+. different lines should be reported

* then press routing button

Pretty sure the last commit fixes this one.

@burmancomp

Copy link
Copy Markdown
Contributor

Pretty sure the last commit fixes this one.

Yes, excellent.

Just one fix left.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

To make sure we understand each other, which one is left?

@burmancomp

Copy link
Copy Markdown
Contributor

Focusing objects with routing buttons when possible as @irrah68 has commented.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I think I'm convinced now.

@burmancomp

Copy link
Copy Markdown
Contributor

I think I'm convinced now.

Thank you very much @LeonarddeR

I can now close #15044 because it is not needed anymore.

@LeonarddeR LeonarddeR changed the title Alternative approach for moving the system caret when routing review cursor Move the system caret when routing review cursor Jun 30, 2023
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I think it should be done now. I probably need to improve the documentation a bit.

@burmancomp

Copy link
Copy Markdown
Contributor

I'm about to test in the morning. @irrah68 will surely test as well. Thank you again.

@irrah68

irrah68 commented Jun 30, 2023

Copy link
Copy Markdown

Thank you very much. This is a significant improvement in NVDA's Braille support.

Object navigation seems to work very well now.

I seem to have some problems when I try to copy text to the clipboard from Web pages. Perhaps you can reproduce this situation.

  1. Go to https://www.nvaccess.org/ and press 2 (or h) to get to the heading that says "We believe that every Blind + Vision Impaired person Deserves the right to freely & easily access a computer!"

  2. Read forward using your display's navigation keys. When you see the text saying "WE CREATE THE SOFTWARE...", press the routing button on W.

  3. Press Shift+End and Ctrl+C to copy the text "WE CREATE THE SOFTWARE WHICH MAKES THAT POSSIBLE" to the clipboard.

I haven't managed to copy this text from the beginning of the selected line. But if I press a routing button at the end of the line, on letter E and then press Shift+Home Ctrl+C, I'll get the text:
WE CREATE THE SOFTWARE WHICH MAKES THAT POSSIBL
Letter E is cut off.

The problem is that I can't copy from the beginning of the line. It always copies the text which says "We believe that every Blind + Vision Impaired person Deserves the right to freely & easily access a computer!". I've tried this for many times without success.

@burmancomp

Copy link
Copy Markdown
Contributor

Thank you very much. This is a significant improvement in NVDA's Braille support.

Object navigation seems to work very well now.

I seem to have some problems when I try to copy text to the clipboard from Web pages. Perhaps you can reproduce this situation.

1. Go to https://www.nvaccess.org/ and press 2 (or h) to get to the heading that says "We believe that every Blind + Vision Impaired person Deserves the right to freely & easily access a computer!"

2. Read forward using your display's navigation keys. When you see the text saying "WE CREATE THE SOFTWARE...", press the routing button on W.

3. Press Shift+End and Ctrl+C to copy the text "WE CREATE THE SOFTWARE WHICH MAKES THAT POSSIBLE" to the clipboard.

The problem is that I can't copy from the beginning of the line. It always copies the text which says "We believe that every Blind + Vision Impaired person Deserves the right to freely & easily access a computer!". I've tried this for many times without success.

The problem seems to be with most left routing button.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

Does this latter problem also apply when braille is explicitly tethered to focus? If so, the bug you're describing isn't related to this pr and should be handled separately.

@LeonarddeR

LeonarddeR commented Jul 1, 2023

Copy link
Copy Markdown
Collaborator Author

... But if I press a routing button at the end of the line, on letter E and then press Shift+Home Ctrl+C, I'll get the text: WE CREATE THE SOFTWARE WHICH MAKES THAT POSSIBL Letter E is cut off.

This is actually expected behavior. Shift+home selects everything before the cursor, not the character at the cursor. If you'd like to select the letter e as well, you'd have to route to the position one character after that letter e.

@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit 78ab159440

@burmancomp

Copy link
Copy Markdown
Contributor

I just added unit tests that should pretty closely mimic behavior of browse mode, and these tests pass. I must be missing something.

@LeonarddeR with nvda_snapshot_pr15064-28601,78ab1594.exe routing in browse mode ("move system caret when routing review cursor" is "always") works for me! First routing button press moves cursors if review cursor is not yet there and when it is already there it activates (depending on element). Second press activates (depending on element). I have not found any new issues.

I am unsure why this snapshot works but previous did not.

I am quite a beginner but interested in learning more so could you @LeonarddeR give some more explanation about implementation for example in "description of developer approach" paragraph.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@LeonarddeR with nvda_snapshot_pr15064-28601,78ab1594.exe routing in browse mode ("move system caret when routing review cursor" is "always") works for me! First routing button press moves cursors if review cursor is not yet there and when it is already there it activates (depending on element). Second press activates (depending on element). I have not found any new issues.

Ah that's great, so seems like the unit tests are valid after all.

I am unsure why this snapshot works but previous did not.

May be it accidentally wasn't the most recent one at time of testing.

I am quite a beginner but interested in learning more so could you @LeonarddeR give some more explanation about implementation for example in "description of developer approach" paragraph.

I'll try to do that later this week.

Comment thread user_docs/en/userGuide.t2t Outdated
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit ce602b7366

@burmancomp

burmancomp commented Jul 6, 2023

Copy link
Copy Markdown
Contributor

@LeonarddeR just another minor change in userGuide.t2t: (comment)

@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit 7d5afa0325

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I tried to clarify the development approach in the initial comment of the issue.

@burmancomp

Copy link
Copy Markdown
Contributor

I tried to clarify the development approach in the initial comment of the issue.

Thank you. I ask if I cannot understand despite of efforts.

As a reminder: should "to focus" be replaced with "to review":

This option is shown only if "[tether braille #BrailleTether]" is set to "Automatically" or "To focus".

Comment thread user_docs/en/userGuide.t2t Outdated
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

Thanks @burmancomp fixed that now.

@LeonarddeR LeonarddeR marked this pull request as ready for review July 8, 2023 07:52
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit 031cb5d553

@burmancomp

Copy link
Copy Markdown
Contributor

Thanks @burmancomp fixed that now.

I suppose @seanbudd should approve this pull request. Hopefully he would have time to review this soon. And hopefully we @LeonarddeR can also co-operate with other issues in the future.

@seanbudd seanbudd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @LeonarddeR - changes generally look good. I have some questions surrounding potential improvements for documentation

Comment thread source/braille.py Outdated
Comment thread source/gui/settingsDialogs.py Outdated
Comment thread source/gui/settingsDialogs.py Outdated
Comment thread tests/unit/test_braille/test_displayTextForGestureIdentifier.py Outdated
Comment thread tests/unit/test_braille/test_displayTextForGestureIdentifier.py Outdated
Comment thread tests/unit/test_cursorManager.py Outdated
Comment thread user_docs/en/userGuide.t2t Outdated
Comment thread user_docs/en/userGuide.t2t Outdated
Comment on lines +1630 to +1633
This setting determines if the system caret should be also tried to be moved
with a routing button press when [braille tethering #BrailleTether] is set to automatic
and braille is temporarily tethered to review, or tethering is explicitly set to the review cursor.
When the object under the review cursor has no physical caret, it will be focused instead.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alternatively, what do you think about deleting these sentences and instead explaining each of the options in more detail?
I think its easier to understand and describe on an option level.

Comment thread user_docs/en/userGuide.t2t Outdated
LeonarddeR and others added 3 commits July 10, 2023 19:17
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@seanbudd thanks for your comment. I think they're all fixed now.

@seanbudd seanbudd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @LeonarddeR

@seanbudd seanbudd merged commit c6bdc66 into nvaccess:master Jul 11, 2023
@nvaccessAuto nvaccessAuto added this to the 2023.2 milestone Jul 11, 2023
@LeonarddeR LeonarddeR deleted the brailleRouteReviewCursorAndSystemCaretAlternative branch August 23, 2025 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

7 participants