-
Notifications
You must be signed in to change notification settings - Fork 29.8k
Add spell check TextSpan creation logic that doesn't rely on composing region #123481
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
justinmc
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall the approach in _buildSubtreesWithoutComposingRegion looks good to me, but two things I'm wondering about:
- Is it also responsible for drawing the composing region? Say if you use a Chinese composing IME on Android with Gboard, will it get routed through _buildSubtreesWithoutComposingRegion and the composing region won't draw? Or maybe this is handled somewhere and there's nothing to worry about.
- Depending on the answer to the above, it feels like we might be able to share a lot of the code between _buildSubtreesWithComposingRegion and _buildSubtreesWithoutComposingRegion. It seems like the fundamental difference between these two methods is just how we detect the word that's currently being edited. Maybe there is more about drawing the composing region that needs to be different, though.
| return TextSpan( | ||
| // We will draw the TextSpan tree based on the composing region, if it is | ||
| // available. | ||
| final bool shouldConsiderComposingRegion = defaultTargetPlatform == TargetPlatform.android; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a comment here describing the fact that Gboard is what considers the composing region, and other Android keyboards may not be compatible with this.
Also we should try using spell check with a newer Samsung device and the Samsung keyboard and see what happens. It won't work, but ideally it would fail silently or maybe even log a warning rather than crash.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I asked about TargetPlatform.android including Samsung because I don't understand why this would crash? Does the Samsung keyboard not have a composing region? If so, I didn't realize that.
I still need to test this, but it should not be a problem because in addition to this check, I can just check if the composing region is valid before using the method that considers it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does have a composing region, it just doesn't use it for the current editing word (as far as I know). Maybe I'm too pessimistic. I thought the Samsung keyboard wasn't compatible at the engine level or something. If it works then that's great.
|
|
||
| return TextSpan( | ||
| style: style, | ||
| children: _buildSubtreesWithoutComposingRegion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What exactly happens if you use this method on Android with Gboard?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just tried it: It works the same, but the composing region doesn't draw. I think that's expected behavior!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes me think that we could use only this method and get rid of the other one. It could remove the composing region when it's part of the replaced word and then draw it. Maybe in another PR if that's going to delay this one, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I definitely think that they can be merged into one and is worth pursuing, then. I am running short on time, though, so unless you're open to playing around with it, I say we delay it for now and file an issue!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm on board with filing an issue for later 👍
It's not responsible for drawing the composing region at all, but it does ignore the word the cursor is touching. However, if it's running on an Android device, if won't call that method, so the problem you described in # 1 should not be a problem, unless I'm misunderstanding what you're asking. Nonetheless, we can probably share some code between the two, although detecting the cursor index to ignore the misspelled word is much simpler logic than dealing with the composing region, so I think some readability would be lost. I would rather not spend time on this now though since I have limited time and I want to focus on getting this landed in time, unless you think it's high priority. Just lmk! |
|
Sounds good, as long as the composing region is being drawn the the code reuse thing is secondary. |
Okay. Does |
|
Yes, a Samsung Android device will have |
Okay, gotcha. I will make sure to test this on one today. |
| // Adjust offset to reflect the difference between where currentSpan | ||
| // was positioned in resultsText versus where the modified version | ||
| // is positioned in newText. | ||
| offset = (adjustedSpanEnd - adjustedSpanStart) - (currentSpan.range.end - currentSpan.range.start) + (adjustedSpanStart - currentSpan.range.start); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The one thing I still want to think about a little more is whether or not to calculate offset this way. The rest of the method assumes really only that one modification has been made between resultsText and newText, and while it may account for more cases than that, this explicitly allows for two operations: modification and movement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thinking about this more, this algorithm covers the following cases:
- One or more
SuggestionSpanshifts, e.g. "Hlello, wrold" --> "Hlello, beautiful wrold" or --> "Oh! Hlello, big, beautiful wrold"* - One or more
SuggestionSpanmodifications, e.g. "Hello, wrold" --> "Hello, wroldd" or --> "Helllo, wroldd" - One or more
SuggestionSpanshifts or modifications, e.g. "Hello, wrold" --> "Hello, big wroldd"
This should be all of the cases that we need to consider because we only care about finding the SuggestionSpan text coming from the old text (resultsText) in the new text, which means that we only care about shifts and modifications of that SuggestionSpan text. Calculating the offset this way in this case, means that we catch case 2 and 3. Case 1 is caught above in the code by adjusting offset based on the actual foundIndex of the SuggestionSpan text. Thus, the code does not at all assume that only one modification has been made, and I believe it is fine as is. I still think multiple modifications/shifts is unlikely, but it doesn't hurt to cover that case.
*Note that all my examples use insertions but deletions for shifts should be accounted for, as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How are you able to do cases 2 and 3? If I'm understanding correctly, those show that a misspelled word has been modified (wrold to wroldd), and you're still able to correct it (world)? That seems super tricky if I'm understanding it correctly. It might be better to not handle those at all. What if the modification creates a correctly spelled word (worl to worlds)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not quite -- case 2 means that a word like wrold has been modified to wroldd, and it would correct the
SuggestionSpan of that word from SuggestionSpan(TextRange(n, n+5), <String>["world"]) to SuggestionSpan(TextRange(n, n+6), <String>["world"]), for example. It's this case (and 3 because 3 involves 2) that pushed me to change this method because if you are typing faster than the spell check results update, the red underline flashes under the misspelled word because of the lag (the cursor index/base offset would not be considered as touching the misspelled word you are composing because an "unknown" addition to the word has been made). My changes fix this issue in the case you described. You would still be compoinsg worl --> worlds, but none of it would be underlined in red, unless after composing, it is still misspelled.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I think I got it, we definitely do want the ability to tell that the word is still being edited in these cases so we can hide the red underline.
| return TextSpan( | ||
| // We will draw the TextSpan tree based on the composing region, if it is | ||
| // available. | ||
| final bool shouldConsiderComposingRegion = defaultTargetPlatform == TargetPlatform.android; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does have a composing region, it just doesn't use it for the current editing word (as far as I know). Maybe I'm too pessimistic. I thought the Samsung keyboard wasn't compatible at the engine level or something. If it works then that's great.
|
|
||
| return TextSpan( | ||
| style: style, | ||
| children: _buildSubtreesWithoutComposingRegion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes me think that we could use only this method and get rid of the other one. It could remove the composing region when it's part of the replaced word and then draw it. Maybe in another PR if that's going to delay this one, though.
| /// Adjusts spell check results to correspond to [newText] if the only results | ||
| /// that the handler has access to are the [results] corresponding to | ||
| /// [resultsText]. | ||
| /// | ||
| /// Used in the case where the request for the spell check results of the | ||
| /// [newText] is lagging in order to avoid display of incorrect results. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you remove this comment? I think we need something explaining what _correctSpellCheckResults is used for.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was an accident! Thanks!
| // Adjust offset to reflect the difference between where currentSpan | ||
| // was positioned in resultsText versus where the modified version | ||
| // is positioned in newText. | ||
| offset = (adjustedSpanEnd - adjustedSpanStart) - (currentSpan.range.end - currentSpan.range.start) + (adjustedSpanStart - currentSpan.range.start); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How are you able to do cases 2 and 3? If I'm understanding correctly, those show that a misspelled word has been modified (wrold to wroldd), and you're still able to correct it (world)? That seems super tricky if I'm understanding it correctly. It might be better to not handle those at all. What if the modification creates a correctly spelled word (worl to worlds)?
| correctedSpellCheckResults.add(adjustedSpan); | ||
| correctedSpellCheckResults.add(adjustedSpan); | ||
| break; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any thoughts on the performance of this algorithm? It seems like it repeatedly uses substring in order to reduce the search space (I think substring would be O(1) and therefore a valid optimization) and then indexOf to search, which I would think is O(reduced search space) in the worst case. Probably reasonable given that the misspelled words could have moved anywhere in the string, but let me know if I'm missing anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do believe that's right and I had the same concern. I think some optimizations could be made if we make further assumptions about when this method will be needed, e.g. if we assume only one modification was made, then we'd only need to do one search and then use lookups to verify the rest of the results based on the first (and only, in this case) calculated offset and searchStart. I think it is worth considering, my only concern here is time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good, it should be fine as-is for now. We haven't done a lot of work on optimizing for large amounts of text in general in Flutter yet.
Also we probably couldn't ever assume that there was only one modification, even though that's probably the case 99% of the time (because the user pressed a single key on their keyboard). It's possible that the IME has sent us any wild modification or even an entirely new string. We encountered the same problem in @Renzo-Olivares's deltas work.
|
I tried this out on iOS and overall it looks great, it feels like a huge upgrade to our iOS text input experience. I found three issues, most probably aren't related to this PR. If not then we should probably just create separate issues for them.
Screen.Recording.2023-04-04.at.10.52.18.AM.mov
Screen.Recording.2023-04-04.at.10.53.12.AM.mov
Screen.Recording.2023-04-04.at.10.58.05.AM.mov |
|
Alright now I've tested it on the Android emulator too. None of the problems above happen on Android (Gboard).
Screen.Recording.2023-04-04.at.11.10.55.AM.mov
Screen.Recording.2023-04-04.at.11.16.02.AM.mov |
|
@justinmc For iOS: 1/2: These are valid issues because the code relies on spaces a lot. I'll take a look today, but worst case, let's file bugs before this makes it to stable. 3: This is expected! Although, iOS must have done an update, because I now see that for misspelled words, the blue highlight is red. May need to file an issue for this before this reaches stable, as well. For Android: 1: This is probably an issue on the framework side, too. I echo what I said about filing an issue! 2: I'm not 100% sure I knew it was supposed to do this. I do see it on native, though. It's a simple fix, so I'll also look at that today. |
|
@camsim99 Are you aware that on master spell check suggestions sometimes appear blank on iOS? No worries because it's fixed in this PR. However, that's a strong reason to make sure this PR gets shipped. |
|
I tried to reproduce all of the numbered bugs I posted above again on master, and they all either can still be reproduced or are blocked by the "blank suggestion" bug I screenshotted. So, this PR does not appear to introduce any new bugs and is an improvement across the board. I agree that we should just file issue for all of the bugs above and move forward with this PR. No worries about fixing Android bug number 2 here if it's not super simple. |
Yes, this is because you are using |
|
Oh right, I keep forgetting that. Thanks! |
justinmc
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor nits on the tests, otherwise looks good.
| TextSpan( | ||
| style: composingTextStyle, | ||
| text: text.substring(composingRegion.start, composingRegion.end) | ||
| text: text.substring(composingRegion.start, composingRegion.end), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good find with all the commas!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Learning slowly!! :)
| () { | ||
| testWidgets( | ||
| 'buildTextSpanWithSpellCheckSuggestions ignores composing region when composing region out of range', | ||
| (WidgetTester tester) async { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why make all of these testWidgets instead of test? I think if it doesn't use tester it should just be a test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did this to use TargetPlatformVariant
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I didn't realize you couldn't use it with test! Sounds good then.
| expect(textSpanTree, equals(expectedTextSpanTree)); | ||
| }); | ||
| expect(textSpanTree, equals(expectedTextSpanTree)); | ||
| }, variant: const TargetPlatformVariant(<TargetPlatform>{ TargetPlatform.android, TargetPlatform.iOS })); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did these tests fail or you just changed them to mobile-only since that's what spell check supports right now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean we can have them run on all, but mobile is what matters so I'd figure I'd change it just to be sure we are testing what we support. Your call! I need them to at least run on those too to try the two different ways of building the TextSpan tree with spell check results.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I say keep it as you have it 👍 . I've seen unnecessary failures happen because a test was running on a platform that it was irrelevant for.
| const String text = 'Hello, wrold! Hey'; | ||
| const String resultsText = 'Hello, wrold!'; | ||
| const TextEditingValue value = TextEditingValue( | ||
| text: text, composing: TextRange(start: 14, end: 17)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this case with a composing region tested somewhere else? If not, maybe we should keep both, one original with the composing region and one your update without composing region?
Same for other tests where the composing region was removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am open to this, but I don't think it's necessary. Cases with the composing region are tested in the other tests, and the purpose of the tests where I removed it have to do with correcting the spell check results versus correctly styling the composing region.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok sounds good to keep it like this if it's covered elsewhere 👍
Just reverted logic to extend For iOS, 1/2: Seems like these are a direct result of the actual spell check results we get back from iOS. Respectively, they do seem to include the semicolon and do not seem to include the apostrophe in the results (at least in that moment while composing the word, e.g "hlelo'" does not have results that include the apostrophe but "hlelo's" does). May have to correct this on the framework end. 3: Native behavior. For Android, 1: This revert fixes this issue! 2: This is still an issue we will need to address on the framework side. |
justinmc
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍. Just a few nits.
| final SuggestionSpan adjustedSpan = SuggestionSpan( | ||
| TextRange( | ||
| start: currentSpan.range.start + offset, end: searchStart), | ||
| currentSpan.suggestions | ||
| start: currentSpan.range.start + offset, end: currentSpan.range.end + offset), | ||
| currentSpan.suggestions, | ||
| ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: I would format it like this:
final SuggestionSpan adjustedSpan = SuggestionSpan(
TextRange(
start: currentSpan.range.start + offset,
end: currentSpan.range.end + offset,
),
currentSpan.suggestions,
);| TextRange(start: adjustedSpanStart, end: adjustedSpanEnd), | ||
| currentSpan.suggestions, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Outdent these two lines.
| () { | ||
| testWidgets( | ||
| 'buildTextSpanWithSpellCheckSuggestions ignores composing region when composing region out of range', | ||
| (WidgetTester tester) async { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I didn't realize you couldn't use it with test! Sounds good then.
| expect(textSpanTree, equals(expectedTextSpanTree)); | ||
| }); | ||
| expect(textSpanTree, equals(expectedTextSpanTree)); | ||
| }, variant: const TargetPlatformVariant(<TargetPlatform>{ TargetPlatform.android, TargetPlatform.iOS })); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I say keep it as you have it 👍 . I've seen unnecessary failures happen because a test was running on a platform that it was irrelevant for.
| const String text = 'Hello, wrold! Hey'; | ||
| const String resultsText = 'Hello, wrold!'; | ||
| const TextEditingValue value = TextEditingValue( | ||
| text: text, composing: TextRange(start: 14, end: 17)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok sounds good to keep it like this if it's covered elsewhere 👍
| expect(textSpanTree, equals(expectedTextSpanTree)); | ||
| }); | ||
| }, variant: const TargetPlatformVariant(<TargetPlatform>{ TargetPlatform.android, TargetPlatform.iOS })); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happened to the test that you added in 7904ab2?
|
Frob is having issues and the branch cut is today. Merging to get this in prior. |
…g region (flutter#123481) Fixes flutter#119534 by adding extra logic to allow the proper `TextSpan` trees to be build with spell check results that does not rely on the composing region. Also, refreshes the algorithm used to correct spell check results to improve cases where spell check results are out of date, but the composing region line does not help smooth over this fact by giving the framework extra time to update while a word is being composed (because a composing region line would cover up any flashing red underlines). ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [Features we expect every widget to implement]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat

Fixes #119534 by adding extra logic to allow the proper
TextSpantrees to be build with spell check results that does not rely on the composing region.Also, refreshes the algorithm used to correct spell check results to improve cases where spell check results are out of date, but the composing region line does not help smooth over this fact by giving the framework extra time to update while a word is being composed (because a composing region line would cover up any flashing red underlines).
Pre-launch Checklist
///).