Skip to content

Improve scrolling experience in browse mode#9919

Closed
LeonarddeR wants to merge 21 commits into
nvaccess:masterfrom
BabbageCom:UIAScroll
Closed

Improve scrolling experience in browse mode#9919
LeonarddeR wants to merge 21 commits into
nvaccess:masterfrom
BabbageCom:UIAScroll

Conversation

@LeonarddeR

Copy link
Copy Markdown
Collaborator

Link to issue number:

Fixes #7075
Closes #6382
Should fix #904

Summary of the issue:

There are several scrolling issues in browse mode, particularly in Edge, where the screen doesn't scroll along with browse mode.

Description of how this pull request fixes the issue:

This pr does the following:

  1. It brings a scrollIntoView method to textInfos. There are several implementations for this, particularly UIA, IA2, Acrobat Object Model and Word object model.
  2. Browse mode calls scrollIntoView on the text info, and if that fails, it tries the current object as a fallback. However, on the object level, we do not support the allignToTop parameter whereas on the text info level, we do.
  3. For most implementations, it makes sure that text will only scroll when the browse mode cursor gets off screen.

Testing performed:

Many tests have been performed by @k-kolev1985 as part of #7075, but I have changed some stuff since then. In any case, this requires testing by sighted people.

Known issues with pull request:

In internet Explorer, we do not support the onlyWhenVisible parameter for scrollIntoView. This is because if we do, navigating with browse mode in Internet Explorer slows down significantly.

Change log entry:

@LeonarddeR LeonarddeR requested a review from Qchristensen July 12, 2019 13:19
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@codeofdusk: It might be good to see what this scrolling code does in UIA consoles. Note that you will still have to trigger it manually though, as the review cursor commands do not yet scroll.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

Here is a try build.

@codeofdusk

Copy link
Copy Markdown
Contributor

@codeofdusk: It might be good to see what this scrolling code does in UIA consoles. Note that you will still have to trigger it manually though, as the review cursor commands do not yet scroll.

Trying to scroll a textInfo into view that is beyond the first visible range (by calling its scrollIntoView method in the Python console) makes review unusable until the caret moves/new text is added. Knowing if text actually scrolls visually would require sighted input. Cc @feerrenrut

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@codeofdusk: Could you elaborate on what you mean with makes review unusable?

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@JulienCochuyt: You might be interested in this pr, since it will drastically improve scrolling behavior in web browsers for sighted users.

@Qchristensen

Copy link
Copy Markdown
Member

Using the About NV Access page (the address has changed from the original issue, but I think it's still the same page, and it does meet the criteria of having a large block of non-linked text): https://www.nvaccess.org/about-nv-access/

With NVDA 2019.2, both Firefox (68.0.2) and Chrome (76.0.3809.100), both scroll as I press the down arrow to move through text. Neither they, nor Edge scroll when using say all however.

Using this PR build, Edge now scrolls when I down arrow through the document, where it doesn't with 2019.2. Neither of the other browsers scroll with down arrow now, and none of them scroll during say all.

@feerrenrut can you confirm either my findings, or that I've gone mad?

@LeonarddeR

LeonarddeR commented Aug 21, 2019 via email

Copy link
Copy Markdown
Collaborator Author

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

Ah, I found it. Will try to come up with an appropriate fix. It also seems that using NVDA+delete is appropriate to test this nonvisually.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

It turns out that IAccessibleTextObject.scrollSubStringTo fails in some cases, that is, it returns S_OK but still does not scroll. Pretty annoying, since it indeed causes a regression. This means that we could only scroll at the object level.

@MarcoZehe

Copy link
Copy Markdown
Contributor

Is this a bug in Firefox, or do you see this in Chrome as well?

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@Qchristensen Would you be able to retest this with the newest build?

note that say all is expected not to work. This is covered in #9937. I could consider putting this in a separate pull request, but I had to change some logic as part of #9937, so it made sense to do that now.

@LeonarddeR LeonarddeR marked this pull request as ready for review August 22, 2019 07:25
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I think this should also fix #5881. Not sure about #6673, though, it might even make it worse.

@JulienCochuyt

JulienCochuyt commented Oct 1, 2019

Copy link
Copy Markdown
Contributor

about #6673, though, it might even make it worse.

This would go beyond my imagination, as the current effect is already quite drastic, especially when you simultaneously attempt to circumvent it using the mouse scroll wheel.

@JulienCochuyt

Copy link
Copy Markdown
Contributor

@LeonarddeR thanks for all the details.
Thus, I guess we are attempting here to alignToTop whenever moving from a visible object to an upper invisible object and alignToBottom when moving to a lower invisible object.
I consider here the origin of the move rather than the attempted direction on purpose: Eg. depending on the document structure, hitting the down arrow might move the virtual caret somewhere up.

@LeonarddeR LeonarddeR added the BabbageWork Pull requests filed on behalf of Babbage B.V. label Oct 8, 2019
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@JulienCochuyt: sorry for me taking some time to get back to this.

Thus, I guess we are attempting here to alignToTop whenever moving from a visible object to an upper invisible object and alignToBottom when moving to a lower invisible object.

Exactly that.

I consider here the origin of the move rather than the attempted direction on purpose: Eg. depending on the document structure, hitting the down arrow might move the virtual caret somewhere up.

Could you elaborate with an example? Hitting down arrow while the caret is at the bottom should idially scroll the new text into view, leaving the caret at the bottom of the view port.

@JulienCochuyt

Copy link
Copy Markdown
Contributor

Could you elaborate with an example? Hitting down arrow while the caret is at the bottom should idially scroll the new text into view, leaving the caret at the bottom of the view port.

If you are browsing content presented in columns by any means, when moving down from the last line of a given column, your destination is the first line of the next column, thus alignToTop in this case.
A specific example is when the canva defines a region between the main content and the footer, visually anchored at the upper right corner of the main content.

@LeonarddeR

LeonarddeR commented Oct 17, 2019 via email

Copy link
Copy Markdown
Collaborator Author

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I wrote this pull request on behalf of @BabbageCom. As I'm leaving @BabbageCom after the 29th of November, I can no longer afford maintaining this pr other than applying very basic review actions. If this pull request requires major changes, they will have to be applied by someone else, e.g. @JulienCochuyt, @sjfbol or whoever else is willing to take it.

@JulienCochuyt

Copy link
Copy Markdown
Contributor

@LeonarddeR As promised, I'll take care of this one (unless of course if @sjfbol is willing to)
I must concentrate on other projects right now with the upcoming of 2019.3, but this is definitely in our pipeline.
I guess you can leave this PR open for reference for now. When I'll return on polishing your code, I'll mark the new PR as superseding this one and closing it.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

It's unlikely that @sjfbol will jump on this pr in the near future, so it would probably best if you do as suggested. I will Finnish this off with fixing the merge conflicts and then I'll leave this open.

@feerrenrut feerrenrut added component/vision Framework for assisting users with low vision enhancement ux labels Apr 8, 2020
@LeonarddeR LeonarddeR marked this pull request as draft May 2, 2020 14:15
@LeonarddeR LeonarddeR added the Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. label May 2, 2020
@lukaszgo1

Copy link
Copy Markdown
Contributor

As in the current Alphas fast browse mode became the default what effect would it have on this PR? Is it even possible to have working scrolling for partially sighted users and do not move focus in browse mode at the same time?

@JulienCochuyt

Copy link
Copy Markdown
Contributor

@lukaszgo1 wrote:

As in the current Alphas fast browse mode became the default what effect would it have on this PR? Is it even possible to have working scrolling for partially sighted users and do not move focus in browse mode at the same time?

As promised, I've started work on taking back this PR, but it requires a huge lot of testing for corner cases.
Theoretically, fast browse mode should not interfere as the focused object can always be safely scrolled out.
Still, thanks for drawing attention on this use case to include in the test suite.

@codeofdusk

Copy link
Copy Markdown
Contributor

@JulienCochuyt Any updates on the testing of this PR?

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

As it seems that this pr hasn't gained much attention by potential takers. Furthermore, when one takes over a pr, it is a common pattern that the old pr is closed in favour of a new one.

I'm closing this pull request to clean up the backlog, happy for anyone to reopen a pr based on this work if desired.

@Adriani90

Copy link
Copy Markdown
Collaborator

I think it would be very beneficial to have this work done, especially for visually impaired people and when working together with sighted people.
Given Internet explorer is discontinued, we don't need to set priority on IE anymore.
However, MsHTML is still being used in several places but nevertheless this PR would be a significant improvement to the current situation anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. BabbageWork Pull requests filed on behalf of Babbage B.V. closed/needs-new-author A new author is required to continue work on the PR component/vision Framework for assisting users with low vision enhancement ux

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When performing a say all on webpages, the screen should scroll along with the speech Content doesn't scroll in Edge Visual issues in browse mode

8 participants