feat(web): previewing gestures - common phone, tablet styling + flick animations 🐵 by jahorton · Pull Request #9825 · keymanapp/keyman

jahorton · 2023-10-23T03:36:55Z

In order to provide visual feedback for ongoing gestures, this PR adds a "gesture preview host" - an element that may be overlaid upon a base key (for tablets) or hosted within the keytip (for phones) in order to provide visual feedback about available & ongoing gestures.

The merged styling does have a mild side effect for tablet layouts. In my experience, the selected key's key-cap tends to move slightly up and left compared to its original position while highlighted. Personally... it actually feels kinda nice. I have looked somewhat into why it's happening and how to resolve it, but there's no obvious straightforward answer due to the differences in styling; it's not worth the effort at this stage of prototyping.

For both cases, during a flick, the preview area will scroll corresponding to the flick motion, bringing any 'destination' flick key-caps into view.

Known possible pain-point

If a flick is started during a modipress, but the modipress key is released before the flick is completed, the current settings will auto-terminate the flick. If we'd prefer, I can enable a 'sustain' mode for it that would allow the modipress-layer flick to complete instead, similar to what's already in place for longpress-key selection. This would not be subject to any sort of time-limit, though - that would likely take extra work.

User Testing

TEST_FLICKS_BASIC: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the u key.
Drag it in an northwest direction (up and to the left) for at least a key-width.
Release it.
Expected result: ù
Tap the u key again and drag it in a northeast direction for at least a key-width before releasing it.
Expected result: ú (does not replace the previous output)
Tap u again and drag north (straight up) in the same manner.
Expected result: û.

TEST_FLICK_CORRECTION: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the u key.
Drag it in an purely west direction (straight left).
Release it.
Expected result: ù
Tap and hold the u key.
Drag it purely south (straight down).
Release it.
Expected result: no output.
Tap and hold the u key again.
Drag it up just a few pixels / millimeters - a very short distance - then release it.
Expected output: u.

TEST_FLICK_ANIMATION: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the u key.
For the following steps, pay attention to the animated preview.
Drag it purely north (straight up) - you should see the û key in the preview area (assuming it's not blocked by your finger).
- If the animation was "jumpy" - if there was a lack of motion at the same time you moved your finger, then a big "jump" to catch up - FAIL this test.
- If any such "jumps" were directly tied to you moving your finger quickly, do not fail this test.
Drag your finger in a big clockwise motion - you should also see previews for ú, and eventually ù, as you do so.
- Again, if the animation felt "jumpy" and not well connected to your input, FAIL this test.
While the ù is visible, lift your finger and end the gesture.
Verify that a ù is output.
Repeat steps 2 through 6 for the a key; you should see similar previews and text there. (à, â, á)

TEST_FLICK_DURING_MODIPRESS: Using the standard "Test unminified KeymanWeb" test page from a mobile device...

Select "English...", then "Gesture Prototyping".
Tap and hold the SHIFT key, keeping it stationary for all following steps.
Tap and hold the U key.
For the following steps, pay attention to the animated preview.
Drag it purely north (straight up) - you should see the Û key in the preview area (assuming it's not blocked by your finger).
- If the animation was "jumpy" - if there was a lack of motion at the same time you moved your finger, then a big "jump" to catch up - FAIL this test.
- If any such "jumps" were directly tied to you moving your finger quickly, do not fail this test.
Drag your finger in a big clockwise motion - you should also see previews for Ú, and eventually Ù, as you do so.
- Again, if the animation felt "jumpy" and not well connected to your input, FAIL this test.
While the Ù is visible, lift your finger and end the flick gesture.
Verify that a Ù is output.
Release the SHIFT key (held since step 2).
Verify that the keyboard returns to the default (lowercase) layer.

Testing Resources

https://jahorton.github.io/#for-gesture-testing.

I have put up KMPs for the three gesture-testing keyboards I've developed for this feature branch at the link above. Be sure to check underneath the "For Gesture Testing" header.

keymanapp-test-bot · 2023-10-23T03:36:59Z

User Test Results

Test specification and instructions

✅ TEST_FLICKS_BASIC (PASSED): Tested this PR with the Android Mobile device (Samsung Galaxy A23 5g) and here is my observation: 1 Opened Test unminified KeymanWeb testpage. 2. Set the keyboard to 'English - Gesture Prototyping'. 3. Tapped and hold the 'u' key. Dragged it to 'up and to the left' most direction, then released it. Verified that the expected results ù appear on the screen. 4. Tapped the 'u' key again then dragged it to 'up and to the right' most direction, then released it. Verified that the 'ú' appeared on the text box and it did not replace the previous output) 5. Tapped the 'u' key again and dragged it straight up. Verified that û appears on the text box. Seems to be working as expected. (notes)
✅ TEST_FLICK_CORRECTION (PASSED): 1. In the English - Gesture Prototyping keyboard, tapped and hold the 'u' key. Dragged it to straight left direction. Then, release it. Verified that ù appear on the text box. 2. Tapped and hold the u key, dragged it to straight down direction. Released the key. 3. Verified that no output appears on the text input screen. 4. Tapped and hold the u key again. Dragged it up just a few pixels then released it. 5. Verified that 'u' appears on the text box. (notes)
✅ TEST_FLICK_ANIMATION (PASSED): 1. In the English - Gesture Prototyping keyboard, tapped and hold the 'u' key. Dragged it to straight-up direction and I was able to see û letter in the preview area without any jumpy. 2. Dragged it in a clockwise motion, I was able to see the previews for ú and ù. 3. Lift my finger when the ù letter is visible. Verified tha ù appears on the text box. 4. Tried the same steps for the letter 'a' and I was able to see similar previews à, â, á on the text box. (notes)
✅ TEST_FLICK_DURING_MODIPRESS (PASSED): Retested as per Josh's suggestion in the Android (ver 13) mobile device and here is my observation: 1. Installed Keyman 17.0.200-alpha-test-9825 in the device. 2. Download and install 'gesture_prototyping.kmp' from the https://jahorton.github.io/#for-gesture-testing link. 3. Set "English - Gesture Prototyping" keyboard. 4. Tapped and hold the SHIFT key. Tapped and hold the U key. Dragged it purely straight upward and could able to see Û in the preview area. 5. Dragged my finger in a big clockwise direction and was able to see the previews for Ú and eventually Ù. 6. Lifting my finger after Ù was visible. Verified that a Ù is output in the text screen. 7. Released the SHIFT key. (notes)

Test Artifacts

Android
Developer
iOS
- Keyman for iOS (simulator image)
- FirstVoices Keyboards for iOS (simulator image)
- TestFlight internal PR build version - 17.0.209 (0.9825.9734)
Keyboards
- Test Keyboards
Web
- KeymanWeb Test Home
Windows

…part may scroll

jahorton · 2023-11-01T08:50:22Z

web/src/engine/osk/src/input/gestures/browser/subkeyPopup.ts

+    if(vkbd.isEmbedded) {
+      safeBounds = new PaddedZoneSource(safeBounds, [topPadding * topScalar, 0, 0]);
+    }


Just caught this as an issue in testing - going off the top of the iOS or Android keyboard was terminating subkeys and flicks almost immediately. There will now be much more room before such a thing kicks in.

If we do want that to be fully, 100% disabled... using NaNs on line 183 can do the trick.

…b/gesture-preview-host

jahorton · 2023-11-02T05:15:52Z

Possibly of note: from the filesize profiling system that's been established...

   ├ ../../../build/engine/osk/obj/input/gestures/specsForLayout.js ──────────────────────────────────────── 9.6kb ─── 2.0%
   │  └ ../../../build/engine/osk/obj/visualKeyboard.js
   │     └ ../../../build/engine/osk/obj/index.js
   │        └ ../../../build/app/browser/obj/keymanEngine.js
   │           └ ../../../build/app/browser/obj/release-main.js

The property names used to spec the gesture-model objects aren't exactly abbreviated like they are for keyboards, and there probably are parts I was a bit WET about. Gotta get the gestures correct first before optimizing their definitions, I suppose. But, should the growth in filesize be a concern, this aspect may be a valuable spot to look. There's also some definite overlap between the 'plain multitap' and 'modipress that can turn into multitap' definitions, for example.

For comparison:

   ├ ../../../build/engine/osk/obj/input/gestures/browser/flick.js ───────────────────────────────────────── 2.4kb ─── 0.5%
   │  └ ../../../build/engine/osk/obj/visualKeyboard.js
   │     └ ../../../build/engine/osk/obj/index.js
   │        └ ../../../build/app/browser/obj/keymanEngine.js
   │           └ ../../../build/app/browser/obj/release-main.js

   ├ ../../../build/engine/osk/obj/input/gestures/browser/multitap.js ────────────────────────────────────── 2.2kb ─── 0.5%
   │  └ ../../../build/engine/osk/obj/visualKeyboard.js
   │     └ ../../../build/engine/osk/obj/index.js
   │        └ ../../../build/app/browser/obj/keymanEngine.js
   │           └ ../../../build/app/browser/obj/release-main.js

mcdurdin

LGTM

common/web/gesture-recognizer/src/engine/configuration/viewportZoneSource.ts

web/src/engine/osk/src/keyboard-layout/gesturePreviewHost.ts

mcdurdin · 2023-11-02T05:44:58Z

web/src/engine/osk/src/keyboard-layout/gesturePreviewHost.ts

+    // const hintLabel = this.hintLabel = document.createElement('div');
+    // hintLabel.className='kmw-key-popup-icon';
+    // hintLabel.textContent = keySpec == keySpec.hintSrc ? keySpec.hint : keySpec.hintSrc?.text;
+    // hintLabel.style.fontWeight= hintLabel.textContent == '\u2022' ? 'bold' : '';
+
+    // // Default positioning puts it far too close to the flick-preview bit.
+    // let yAdjustment = 0;
+    // hintLabel.style.marginTop = `-${yAdjustment}px`;
+
+    // // b/c multitap's border forces position shifting
+    // let xAdjustment = 0;
+    // hintLabel.style.marginRight = `-${xAdjustment}px`;
+
+    // base.appendChild(hintLabel);


Commented code here

Right. I kept it around for possible use for multitap previews, should we have the time & space for that. Might have to triage it out and make an issue with reference to this block as a future 'feature request' instead, though.

Removed in #9998 along with some of the cleanup requested for #9909.

web/src/engine/osk/src/visualKeyboard.ts

…b/gesture-preview-host

bharanidharanj · 2023-11-03T11:01:54Z

Test Results

TEST_FLICKS_BASIC (PASSED): Tested this PR with the Android Mobile device (Samsung Galaxy A23 5g) and here is my observation: 1 Opened Test unminified KeymanWeb testpage. 2. Set the keyboard to 'English - Gesture Prototyping'. 3. Tapped and hold the 'u' key. Dragged it to 'up and to the left' most direction, then released it. Verified that the expected results ù appear on the screen. 4. Tapped the 'u' key again then dragged it to 'up and to the right' most direction, then released it. Verified that the 'ú' appeared on the text box and it did not replace the previous output) 5. Tapped the 'u' key again and dragged it straight up. Verified that û appears on the text box. Seems to be working as expected.
TEST_FLICK_CORRECTION (PASSED): 1. In the English - Gesture Prototyping keyboard, tapped and hold the 'u' key. Dragged it to straight left direction. Then, release it. Verified that ù appear on the text box. 2. Tapped and hold the u key, dragged it to straight down direction. Released the key. 3. Verified that no output appears on the text input screen. 4. Tapped and hold the u key again. Dragged it up just a few pixels then released it. 5. Verified that 'u' appears on the text box.
TEST_FLICK_ANIMATION (PASSED): 1. In the English - Gesture Prototyping keyboard, tapped and hold the 'u' key. Dragged it to straight-up direction and I was able to see û letter in the preview area without any jumpy. 2. Dragged it in a clockwise motion, I was able to see the previews for ú and ù. 3. Lift my finger when the ù letter is visible. Verified tha ù appears on the text box. 4. Tried the same steps for the letter 'a' and I was able to see similar previews à, â, á on the text box.

…b/gesture-preview-host

bharanidharanj · 2023-11-10T08:16:57Z

Test Results

TEST_FLICK_DURING_MODIPRESS (PASSED): Retested as per Josh's suggestion in the Android (ver 13) mobile device and here is my observation: 1. Installed Keyman 17.0.200-alpha-test-9825 in the device. 2. Download and install 'gesture_prototyping.kmp' from the https://jahorton.github.io/#for-gesture-testing link. 3. Set "English - Gesture Prototyping" keyboard. 4. Tapped and hold the SHIFT key. Tapped and hold the U key. Dragged it purely straight upward and could able to see Û in the preview area. 5. Dragged my finger in a big clockwise direction and was able to see the previews for Ú and eventually Ù. 6. Lifting my finger after Ù was visible. Verified that a Ù is output in the text screen. 7. Released the SHIFT key.
Verified that the keyboard returns to the default (lowercase) layer. It seems to be working as expected.

..Keyman Version

..Able to see the expected output on the screen

…b/gesture-preview-host

jahorton requested a review from mcdurdin as a code owner October 23, 2023 03:36

keymanapp-test-bot bot added the epic-gestures label Oct 23, 2023

keymanapp-test-bot bot changed the title ~~feat(web): previewing gestures - common phone, tablet styling~~ feat(web): previewing gestures - common phone, tablet styling 🐵 Oct 23, 2023

keymanapp-test-bot bot added this to the A17S24 milestone Oct 23, 2023

github-actions bot added web/ feat labels Oct 23, 2023

jahorton marked this pull request as draft October 23, 2023 03:38

jahorton mentioned this pull request Oct 24, 2023

feat(web): generalized multitap 🐵 #9740

Merged

mcdurdin modified the milestones: A17S24, A17S25 Oct 27, 2023

jahorton added 17 commits October 31, 2023 13:34

feat(web): start of the gesture preview host

28ab0e8

fix(web): minor goof in last commit

58da961

feat(web): prototype flick preview styling

90f9baf

change(web): puts flick previews in multitap ele to prevent overlay

3f9fd8c

feat(web): demo flag, styling nuances, ability to fade invalid entries

55f004f

feat(web): multitap + longpress styling statically positioned, flick …

efa4806

…part may scroll

fix(web): border-rounding for previews

f6d9354

chore(web): post-rebase patchup

1ff8d68

change(web): gesture-preview longevity, causality

5be435b

fix(web): no roaming preview with flicks

9ef0ad3

change(web): trims previous preview styling

db77fa1

feat(web): first scrolling flick preview

3483781

feat(web): 'natural' scrolling, smoother scroll start

e562253

feat(web): flick-scroll clearing

d3ad24e

chore(web): post-rebase patchup

6169ffe

fix(web): flick-scroll only for keys with flicks

bb2250f

fix(web): random viewport-size def bug

1715463

jahorton force-pushed the feat/web/gesture-preview-host branch from 3811f18 to 1715463 Compare October 31, 2023 07:17

jahorton changed the base branch from fix/web/state-change-source-rewrite to feat/web/flick-implementation October 31, 2023 07:17

fix(web): early off-the-top cancellation in app/webview mode

3c9c036

jahorton commented Nov 1, 2023

View reviewed changes

jahorton and others added 2 commits November 2, 2023 10:43

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

00cdbdf

…b/gesture-preview-host

fix(web): app/webview globe-key key-up signalling

1cef89f

mcdurdin approved these changes Nov 2, 2023

View reviewed changes

jahorton added 4 commits November 2, 2023 16:22

fix(web): Android API 21 compat

3e3e6ed

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

01c85b7

…b/gesture-preview-host

fix(web): fixes tablet preview-host cleanup

31257d9

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

874e463

…b/gesture-preview-host

keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Nov 3, 2023

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

ba97cc2

…b/gesture-preview-host

keymanapp-test-bot bot added the user-test-required User tests have not been completed label Nov 6, 2023

jahorton added 5 commits November 6, 2023 08:59

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

8ddb91b

…b/gesture-preview-host

chore(web): applies a few PR suggestions

fab64bc

fix(web): better consistency for flick-animation b/t phone and tablet

1136b83

fix(web): lm-worker null guard

8d6610a

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

f1f0991

…b/gesture-preview-host

jahorton mentioned this pull request Nov 7, 2023

change(web): locked flicks 🐵 #9957

Merged

keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Nov 10, 2023

mcdurdin modified the milestones: A17S25, A17S26 Nov 13, 2023

jahorton added 2 commits November 14, 2023 09:50

chore(web): Merge branch 'feat/web/flick-implementation' into feat/we…

3e3929d

…b/gesture-preview-host

chore(web): a bit more cleanup

b0b696d

Base automatically changed from feat/web/flick-implementation to feature-gestures November 14, 2023 08:26

jahorton merged commit 1a7245e into feature-gestures Nov 14, 2023

jahorton deleted the feat/web/gesture-preview-host branch November 14, 2023 08:26

jahorton mentioned this pull request Dec 8, 2023

bug(web): key previews use incorrect font #9187

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(web): previewing gestures - common phone, tablet styling + flick animations 🐵#9825

feat(web): previewing gestures - common phone, tablet styling + flick animations 🐵#9825
jahorton merged 33 commits intofeature-gesturesfrom
feat/web/gesture-preview-host

jahorton commented Oct 23, 2023 •

edited

Loading

Uh oh!

keymanapp-test-bot bot commented Oct 23, 2023 •

edited

Loading

Uh oh!

jahorton Nov 1, 2023

Uh oh!

jahorton commented Nov 2, 2023

Uh oh!

mcdurdin left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcdurdin Nov 2, 2023

Uh oh!

jahorton Nov 6, 2023

Uh oh!

jahorton Nov 14, 2023

Uh oh!

Uh oh!

bharanidharanj commented Nov 3, 2023

Uh oh!

bharanidharanj commented Nov 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jahorton commented Oct 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Known possible pain-point

User Testing

Testing Resources

Uh oh!

keymanapp-test-bot bot commented Oct 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User Test Results

Test Artifacts

Uh oh!

jahorton Nov 1, 2023

Choose a reason for hiding this comment

Uh oh!

jahorton commented Nov 2, 2023

Uh oh!

mcdurdin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcdurdin Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

jahorton Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

jahorton Nov 14, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bharanidharanj commented Nov 3, 2023

Test Results

Uh oh!

bharanidharanj commented Nov 10, 2023

Test Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jahorton commented Oct 23, 2023 •

edited

Loading

keymanapp-test-bot bot commented Oct 23, 2023 •

edited

Loading