Skip to content

Enhance logic that detects selections using the display model#10040

Closed
LeonarddeR wants to merge 9 commits into
nvaccess:masterfrom
BabbageCom:advancedSelectionCriteria
Closed

Enhance logic that detects selections using the display model#10040
LeonarddeR wants to merge 9 commits into
nvaccess:masterfrom
BabbageCom:advancedSelectionCriteria

Conversation

@LeonarddeR

@LeonarddeR LeonarddeR commented Aug 5, 2019

Copy link
Copy Markdown
Collaborator

Link to issue number:

Related to #7418
Related to #12658
Split from #8414
Base for #9035

Summary of the issue:

The display model is able to detect selected text based on the background and foreground color. Therefore, it asks the system for the selection background and foreground colors and returns offsets in the text based on this info.

However, this information is often not accurate, i.e. because a particular control implements different colors to expose a selection.

To give some real life examples. I've seen several situations in the wild where selection had to be detected based on two different combinations of foreground and background color. There are also cases where the selection can be detected based on only the foreground color.

Description of how this pull request fixes the issue:

I borrowed the idea of property conditions from UIAUtils to create one or more conditions that should be met in order to have selection to be reported. To evaluate a condition, you can simply throw one or more condition dicts at the evaluateCondition method of a textInfos.Field instance.

Long story short, the intention of this pr is to make it easy for add-ons or in built appmodules to customize the logic used to detect seletion changes using the display model.

Testing performed:

Several tests in the wild for corporate applications, though it might be beneficial to write some unitTests for this.

Known issues with pull request:

None

Change log entry:

  • Changes for developers
    • You can now much more easily change the conditions for a DisplayModelTextInfo to report selections.

Code Review Checklist:

  • Pull Request description is up to date.
  • Unit tests.
  • System (end to end) tests.
  • Manual testing.
  • User Documentation.
  • Change log entry.
  • Context sensitive help for GUI changes.
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers

@lukaszgo1

Copy link
Copy Markdown
Contributor

Would this somehow improve #9035? If so perhaps you may consider adding an example app module fixing one of the bugs referenced there as part of this pr?

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@lukaszgo1 commented on 6 aug. 2019 13:02 CEST:

Would this somehow improve #9035? If so perhaps you may consider adding an example app module fixing one of the bugs referenced there as part of this pr?

Yes, it definitely would, though you would still have to subclass the used textInfo and implement the right selectionCondition.

I've experimented with saving the selection condition on the object, but that has its downsides.

Here is an example I"m currently working on it is for a corporate application though:

class KingMenuTextInfo(DisplayModelTextInfo):
	selectionCondition = [{
		'color': [RGB(255, 255, 255)],
	}]

This example treats all white text as being selected.

@LeonarddeR LeonarddeR added the BabbageWork Pull requests filed on behalf of Babbage B.V. label Oct 8, 2019
@lukaszgo1

Copy link
Copy Markdown
Contributor

@feerrenrut Is this on your radar for 2020.3 perhaps? Currently it is required to copy entire _getSelectionOffsets for every Displaymodel textinfo in an appmodule and change just a few lines if the default criteria for selection are not used in a given program.

@feerrenrut feerrenrut self-requested a review July 22, 2020 12:58
@feerrenrut feerrenrut added the Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. label Aug 11, 2020
@lukaszgo1

Copy link
Copy Markdown
Contributor

Hi @feerrenrut I'm quite interested in having this merged sooner than later so if @LeonarddeR can't work on this further I'd be happy to take this. What would help however would be to have this PR reviewed - perhaps it can be approved as is and no further actions are necessary.

@feerrenrut

Copy link
Copy Markdown
Contributor

@lukaszgo1 That would be great! Just reviewing this quickly again, the query idea (Field.evaluateQuery) is complicated, it would be good if it had unit tests.

It seems to me that the intention with this PR is not to change user visible behavior, but to give greater flexibility for specializing behavior for certain apps. If this is correct it would be good to clarify that in the description. Also, it would be good to think about how to verify that the existing behavior hasn't changed.

@michaelDCurran Do you have anything to add to this?

If you wish to take this on, create a new branch based on this one and open a new PR. Comment the PR number here and then I'll close this one.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

the query idea (Field.evaluateQuery) is complicated.

I stole the logic from UIAUtils.createUIAMultiPropertyCondition

It seems to me that the intention with this PR is not to change user visible behavior, but to give greater flexibility for specializing behavior for certain apps. If this is correct it would be good to clarify that in the description.

This is correct.

@feerrenrut

Copy link
Copy Markdown
Contributor

Since this is hard for us to test across a wide range of applications, and there are likely lots of users depending on the current behavior with legacy enterprise applications, I'd like to see this a feature flag wrapping the new behavior.

I'd also like to see some automated tests for this.

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

Since this is hard for us to test across a wide range of applications, and there are likely lots of users depending on the current behavior with legacy enterprise applications, I'd like to see this a feature flag wrapping the new behavior.

There won't be any difference in behaviour with this pr, it only majorly increases the customizability of the selection detection logic.

@feerrenrut

Copy link
Copy Markdown
Contributor

So the intention is that there is no change in behavior, but customizability is easier? Customization via add-ons I assume?

I was thinking if there is a bug, it would be good to be able revert to old code. Though looking at this more, I guess there is only singular logic built into NVDA. There may be bugs exposed by add-ons, but the addon can be disabled.

If the intention here is to make this easy for add-ons or in built appmodules to customize, it would be useful for it to be included in the description.

@LeonarddeR LeonarddeR requested a review from a team as a code owner July 24, 2021 10:59
@LeonarddeR LeonarddeR marked this pull request as draft July 24, 2021 11:00
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit 2673e5fd25

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I'm taking this again.

@LeonarddeR LeonarddeR removed the Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. label Jul 24, 2021
@AppVeyorBot

Copy link
Copy Markdown

See test results for failed build of commit 44ac474905

@lukaszgo1

lukaszgo1 commented Oct 9, 2021

Copy link
Copy Markdown
Contributor

To test this code I've converted the way in which I detect selection in my add-on for Becky! Internet Mail and while it is much easier to provide selection criteria when the given text info uses different colors for showing selection this PR broke NVDA's ability to detect selections spanning multiple lines in DisplayModel based text infos (only first line of the selection is reported when querying for selected text). I haven't investigated further as of yet since @LeonarddeR told us they want to continue working on this.

@LeonarddeR

LeonarddeR commented Oct 11, 2021 via email

Copy link
Copy Markdown
Collaborator Author

@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

I have time nor interest to work on this further. @lukaszgo1 would you be interested to take this further?

@seanbudd seanbudd added the Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. label Nov 10, 2022
@seanbudd

Copy link
Copy Markdown
Member

Closing as abandoned

@seanbudd seanbudd closed this Nov 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Abandoned requested reports or updates are missing since more than 1 year, author or users are not available. BabbageWork Pull requests filed on behalf of Babbage B.V. component/display-model enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants