Fix: Various input issues with input into streamed UE widgets (including IME support) #564
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Discussion result: Yes, we can remove those events. |
🦋 Changeset detectedLatest commit: 61a25a0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
mcottontensor
approved these changes
Apr 7, 2025
… UI bleeding into UE keyboard input
… needed for IME input which is now handled via modal
Mentioned the use of software cursor as an option.
…y to full" This reverts commit b7c72be.
29df26a to
8b77930
Compare
…ependency to full"" This reverts commit 8b77930.
github-actions bot
pushed a commit
that referenced
this pull request
Apr 8, 2025
…ing IME support) (#564) * Add stub of editTextModal to application * Added events for showing onscreen keyboard, hooked them up to an edit text modal * Added explicitly stopping event propogation on text input into stream UI bleeding into UE keyboard input * Added hidden text box input for desktop IME use * Added config option to not show the modal * Added new docs page about typing into streamed inputs * Remove oncomposition event from keyboard controller as they were only needed for IME input which is now handled via modal * Removed now unused styles from PixelStreamingApplicationStyles.ts * Update Typing into streamed text inputs.md Mentioned the use of software cursor as an option. --------- Co-authored-by: Matthew.Cotton <matt@tensorworks.com.au> (cherry picked from commit 5fb6eb6)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
mcottontensor
pushed a commit
that referenced
this pull request
Apr 8, 2025
…ing IME support) (#564) (#575) * Add stub of editTextModal to application * Added events for showing onscreen keyboard, hooked them up to an edit text modal * Added explicitly stopping event propogation on text input into stream UI bleeding into UE keyboard input * Added hidden text box input for desktop IME use * Added config option to not show the modal * Added new docs page about typing into streamed inputs * Remove oncomposition event from keyboard controller as they were only needed for IME input which is now handled via modal * Removed now unused styles from PixelStreamingApplicationStyles.ts * Update Typing into streamed text inputs.md Mentioned the use of software cursor as an option. --------- Co-authored-by: Matthew.Cotton <matt@tensorworks.com.au> (cherry picked from commit 5fb6eb6) Co-authored-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>
mcottontensor
added a commit
that referenced
this pull request
Apr 8, 2025
* Action audit (#565) (#566) - Removing duplicated push and pull_request triggers - Removing multiple install commands when one root will work. - Removing references to UE5.5 in favor of wildcard matching (cherry picked from commit 4ee5695) Co-authored-by: mcottontensor <80377552+mcottontensor@users.noreply.github.com> * [UE5.5] Moved frontend library from a peer dep to a normal dep in ui-library (#568) (#569) * Moved frontend library from a peer dep to a normal dep in ui-library (#568) (cherry picked from commit e255bc0) # Conflicts: # Frontend/ui-library/package.json # package-lock.json * Fixing package-lock issues * Fixing issue where workspace path doesnt exist in npm publish action (#570) (#571) Trying to also fix issues with the changelog update perms. (cherry picked from commit 4f9b8a7) Co-authored-by: mcottontensor <80377552+mcottontensor@users.noreply.github.com> * Trying to fix changelog update action * Still trying to fix changeset update action * What a nightmare * Updated NPM changelogs (#572) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fixing publish action. * Fixing publish action more. * Wrong package file. * Fix: Various input issues with input into streamed UE widgets (including IME support) (#564) (#575) * Add stub of editTextModal to application * Added events for showing onscreen keyboard, hooked them up to an edit text modal * Added explicitly stopping event propogation on text input into stream UI bleeding into UE keyboard input * Added hidden text box input for desktop IME use * Added config option to not show the modal * Added new docs page about typing into streamed inputs * Remove oncomposition event from keyboard controller as they were only needed for IME input which is now handled via modal * Removed now unused styles from PixelStreamingApplicationStyles.ts * Update Typing into streamed text inputs.md Mentioned the use of software cursor as an option. --------- Co-authored-by: Matthew.Cotton <matt@tensorworks.com.au> (cherry picked from commit 5fb6eb6) Co-authored-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com> * Platform scripts audit (#567) (#577) * Limited the bash script to build cjs * Added a cjs build target * Updating bash script to be a little more friendly with builds. * Added rebuild options to bash script * Updating windows batch script to be inline with bash script. * Cleaned up arguments further for the bash script * Tweaking build behaviour. Updating args * Sorting out webpack configs (cherry picked from commit f7abd33) Co-authored-by: mcottontensor <80377552+mcottontensor@users.noreply.github.com> * Updated NPM changelogs (#576) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>
mcottontensor
added a commit
that referenced
this pull request
Apr 8, 2025
* Action audit (#565) (#566) - Removing duplicated push and pull_request triggers - Removing multiple install commands when one root will work. - Removing references to UE5.5 in favor of wildcard matching (cherry picked from commit 4ee5695) Co-authored-by: mcottontensor <80377552+mcottontensor@users.noreply.github.com> * [UE5.5] Moved frontend library from a peer dep to a normal dep in ui-library (#568) (#569) * Moved frontend library from a peer dep to a normal dep in ui-library (#568) (cherry picked from commit e255bc0) # Conflicts: # Frontend/ui-library/package.json # package-lock.json * Fixing package-lock issues * Fixing issue where workspace path doesnt exist in npm publish action (#570) (#571) Trying to also fix issues with the changelog update perms. (cherry picked from commit 4f9b8a7) Co-authored-by: mcottontensor <80377552+mcottontensor@users.noreply.github.com> * Trying to fix changelog update action * Still trying to fix changeset update action * What a nightmare * Updated NPM changelogs (#572) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fixing publish action. * Fixing publish action more. * Wrong package file. * Fix: Various input issues with input into streamed UE widgets (including IME support) (#564) (#575) * Add stub of editTextModal to application * Added events for showing onscreen keyboard, hooked them up to an edit text modal * Added explicitly stopping event propogation on text input into stream UI bleeding into UE keyboard input * Added hidden text box input for desktop IME use * Added config option to not show the modal * Added new docs page about typing into streamed inputs * Remove oncomposition event from keyboard controller as they were only needed for IME input which is now handled via modal * Removed now unused styles from PixelStreamingApplicationStyles.ts * Update Typing into streamed text inputs.md Mentioned the use of software cursor as an option. --------- Co-authored-by: Matthew.Cotton <matt@tensorworks.com.au> (cherry picked from commit 5fb6eb6) Co-authored-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com> * Platform scripts audit (#567) (#577) * Limited the bash script to build cjs * Added a cjs build target * Updating bash script to be a little more friendly with builds. * Added rebuild options to bash script * Updating windows batch script to be inline with bash script. * Cleaned up arguments further for the bash script * Tweaking build behaviour. Updating args * Sorting out webpack configs (cherry picked from commit f7abd33) Co-authored-by: mcottontensor <80377552+mcottontensor@users.noreply.github.com> * Updated NPM changelogs (#576) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Rebuilt package-lock --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relevant components:
Problem statement:
There are various problems with input into streamed UE text inputs currently:
Input of non-latin characters (e.g. Chinese, Japanese, Korean etc.) does not work due to IME assistance/auto-completion requiring a visible (in-focus) text input to be shown in the browser (currently we have a hidden text box).
The native on-device on-screen keyboard is shown but there is no text box to see your edits or submit them. Thus, typing on mobile is non-functional.
Typing into other frontend widgets (e.g. the settings panel) when there is a UE widget focused causes typing to occur into the UE widget and the frontend component.
Copy and pasting input from your clipboard into the streamed UE widget is not possible.
There is no frontend exposed event for when UE sends text input content to the frontend, meaning customisation of behaviour is not possible.
Solution
The solutions to these problems in order are:
Show an on-screen in-browser modal that allows the user to enter text in the browser and use IME assistance.
Showing a modal on the browser side will allow the on-device keyboard to pop up when focus is given to that modal and will mimic expected text input UX on the device. The modal will have a confirm button that when clicked will submit the input. Those allowing both desktop and mobile to signal when they are done typing.
When frontend components are typed into they should
stopPropogationso the input is not forwarded to UE.Copy and paste will work as it normally does with no extra handling required on the browser or UE side when typing into the proposed modal.
We should expose a Pixel Streaming frontend event for when UE text contents is focused and ready for editing, we already receive the
showOnScreenKeyboarddatachannel message, the fix is simply to expose aShowOnScreenKeyboardEventevent related to this message.Demonstration
I have recorded a video of the proposed modal solution showing standard input, IME input, and copy and paste:
add_nonlatin_input.mp4
Documentation
Add: Docs about new config option
UseModalForTextInput(which istrueby default). This option allows users to completely bypass the modal and simple forward all keyboard input back to UE (but this mean text field editing will not work on mobile, IME input will not work, and copy and paste will not work).Add: New docs section about how typing into streamed widgets works in Pixel Streaming.
New docs commit here
Test Plan and Compatibility
I have tested this PR on:
FindFocusedWidgetbeing added toOnMouseDownin this commit.TextboxEntrydatachannel message.closes #544
closes #475
closes #48