Skip to content

feat(web): Caps Layer and double-tap gesture 🕊#5989

Merged
mcdurdin merged 4 commits intomasterfrom
feat/web/3620-caps-layer
Jan 17, 2022
Merged

feat(web): Caps Layer and double-tap gesture 🕊#5989
mcdurdin merged 4 commits intomasterfrom
feat/web/3620-caps-layer

Conversation

@mcdurdin
Copy link
Copy Markdown
Member

@mcdurdin mcdurdin commented Nov 30, 2021

Fixes #3620.

Implements the Caps Lock layer support and the double-tap gesture on the shift key to access it.

The double-tap gesture has been implemented with a view to extension to support other multi-tap gestures in the future. However, for now, it is limited to supporting the Shift key, if and only if the keyboard includes a Caps layer.

The reason for this v15 limitation is that multi-tap on regular keys would involve either rewinding the previous keystroke (the first tap), or forcing keyboard developers to consider 'rota' style rules in their keyboards to support the multi-tap gestures, as we need to make sure that the first tap is accepted and processed for immediate feedback. This needs more design, to avoid unnecessary complexity in the keyboards and/or the rewinding of the keystroke (even though that is conceptually supported in Keyman Engine for Web already). Basically, we don't want to constrain the way that a keyboard author may use the multi-tap gesture by hard-coding the rewind, but neither do we want to make all multi-tap gestures needlessly complex to author.

The shift key (and other modifiers, potentially in future) needs special support for multi-tap as the key that is being tapped changes with the layer change. This is currently managed through recognising K_SHIFT in the key id.

I have tried to follow the PendingGesture pattern for multi-tap, and the gesture itself supports a series of taps, not just a double-tap. The maximum time to complete the tap series is 125msec * number-of-taps, so for a double-tap is 250msec.

The changes to support a Caps Lock layer itself were minimal; just adding the text.KeyboardProcessor.getStateFromLayer function and calling it during KeyEvent construction. The remaining changes relate to the multi-tap gesture.

Minor changes:

  • I moved constructNullKeyEvent to KeyEvent in order to make it more accessible to other classes.
  • The multi-tap gesture does not have a promise to complete, so that is now an optional member of the PendingGesture interface.

Note:

  • The testing page for KeymanWeb includes a test for just the Caps Lock layer support, caps_lock_layer_3620.js, and a full Caps Lock and Start of Sentence test, full_caps_3620_3621.js.

User Testing

SUITE_BASELINE: Test that nothing has changed in existing keyboards

  1. Run through basic usage tests of a variety of keyboards, including sil_euro_latin and khmer_angkor.
  2. Test longpress keys, layer switching, flick-up gesture.
  3. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!
  • TEST_BASELINE_WEB: On testing/unminified, run through the above tests. Make sure no script errors are encountered. Do this both for desktop and touch modes.
  • TEST_BASELINE_ANDROID: In Keyman for Android, on an Android device or emulator, run through the above tests, both system and in-app keyboards.
  • TEST_BASELINE_IPHONE: In Keyman for iPhone, on an iPhone device or emulator, run through the above tests, both system and in-app keyboards.
  • TEST_BASELINE_IPAD: In Keyman for iPad, on an iPad device or emulator, run through the above tests, both system and in-app keyboards.

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

  1. Run through a basic usage test of caps_lock_layer_3620.zip.
  2. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).
  3. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.
  4. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.
  5. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!
  • TEST_CAPS_WEB: On testing/"Test Caps Lock Layer (spec: Caps Lock layer for touch layouts #3620)", run through the above tests. Make sure no script errors are encountered.
  • TEST_CAPS_ANDROID: In Keyman for Android, on an Android device if possible, run through the above tests, both system and in-app keyboards.
  • TEST_CAPS_IPHONE: In Keyman for iPhone, on an iPhone device if possible, run through the above tests, both system and in-app keyboards.
  • TEST_CAPS_IPAD: In Keyman for iPad, on an iPad device if possible, run through the above tests, both system and in-app keyboards.

Fixes #3620.

Implements the Caps Lock layer support and the double-tap gesture on the
shift key to access it.

The double-tap gesture has been implemented with a view to extension to
support other multi-tap gestures in the future. However, for now, it is
limited to supporting the Shift key, if and only if the keyboard
includes a Caps layer.

The reason for this v15 limitation is that multi-tap on regular keys
would involve either rewinding the previous keystroke (the first tap),
or forcing keyboard developers to consider 'rota' style rules in their
keyboards to support the multi-tap gestures, as we need to make sure
that the first tap is accepted and processed for immediate feedback.
This needs more design, to avoid unnecessary complexity in the keyboards
and/or the rewinding of the keystroke (even though that is conceptually
supported in Keyman Engine for Web already). Basically, we don't want to
constrain the way that a keyboard author may use the multi-tap gesture
by hard-coding the rewind, but neither do we want to make all multi-tap
gestures needlessly complex to author.

The shift key (and other modifiers, potentially in future) needs special
support for multi-tap as the key that is being tapped changes with the
layer change. This is currently managed through recognising `K_SHIFT` in
the key id.

I have tried to follow the `PendingGesture` pattern for multi-tap, and
the gesture itself supports a series of taps, not just a double-tap. The
maximum time to complete the tap series is 125msec * number-of-taps, so
for a double-tap is 250msec.

The changes to support a Caps Lock layer itself were minimal; just
adding the `text.KeyboardProcessor.getStateFromLayer` function and
calling it during `KeyEvent` construction. The remaining changes relate
to the multi-tap gesture.

Minor changes:
* I moved `constructNullKeyEvent` to `KeyEvent` in order to make it
  more accessible to other classes.
* The multi-tap gesture does not have a promise to complete, so that is
  now an optional member of the `PendingGesture` interface.
@mcdurdin mcdurdin added this to the A15S19 milestone Nov 30, 2021
@keymanapp-test-bot keymanapp-test-bot bot added the user-test-required User tests have not been completed label Nov 30, 2021
@keymanapp-test-bot
Copy link
Copy Markdown

keymanapp-test-bot bot commented Nov 30, 2021

User Test Results

Test specification and instructions

🟥 SUITE_BASELINE: Test that nothing has changed in existing keyboards

  • TEST_BASELINE_WEB (PASSED): The OSK seems to work fine, but there is a catch. I don't know what "script error" would entail, but as far as the usage of the OSK is concerned, nothing stops me from using the keyboards. (notes)
  • 🟥 TEST_BASELINE_ANDROID (FAILED): ???** In Keyman for Android, on an Android device or emulator, run through the above tests, both system and in-app keyboards. (notes)
  • TEST_BASELINE_IPHONE (PASSED): ?** (notes)
  • TEST_BASELINE_IPAD (PASSED): No obvious error is found. (notes)

🟥 SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

Test Artifacts

Removed the 'heuristic' test as it is probably not appropriate for
testing. Removed the mobile-specific platform from the touch layout as
it did not include a caps layer (and should have been identical to the
tablet one anyway).
@mcdurdin mcdurdin changed the title feat(web): Caps Layer and double-tap gesture 🍆 feat(web): Caps Layer and double-tap gesture 🕊 Dec 9, 2021
@mcdurdin mcdurdin modified the milestones: A15S19, A15S20 Dec 11, 2021
Copy link
Copy Markdown
Contributor

@ermshiperete ermshiperete left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't see anything obvious, but my knowledge in that part of the code is pretty limited 😄

}

/**
* Get state key state from layer id
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the double state intentional?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, on 101 key keyboards, the 'state' keys are the caps lock, num lock, scroll lock keys, but we only care about Caps Lock. So this is retrieving the state of these keys 😆.

Both getStateFromLayer and getModifierState above could be better; right now a layer called "foolscapsymbols" (okay, it's a stretch) would also be considered a 'caps lock' layer.

c this is a heuristic: if we find two+ characters in capitals, leave
c us in shift layer, assuming the user hasn't just switched layer
c themselves. Not sure if this is a good idea!
c if(&layerChanged = "0") if(&layer = 'shift') any(caps) any(caps) > layer('caps') c ??? is this a good idea?
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you want to keep this comment?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps I should probably remove that heuristic altogether? I will leave it for now but think about it for a follow-up PR.

@MakaraSok
Copy link
Copy Markdown

SUITE_BASELINE: Test that nothing has changed in existing keyboards

  • TEST_BASELINE_WEB: PASSED The OSK seems to work fine, but there is a catch. I don't know what "script error" would entail, but as far as the usage of the OSK is concerned, nothing stops me from using the keyboards.

    The OSK doesn't show keycaps in capital letters when Caps key is active.

    On Touch devices, everything works without any issue, but on iPad: the characters on keycaps are kinda a little too small.

  • TEST_BASELINE_ANDROID: FAILED?

  • TEST_BASELINE_IPHONE: PASSED?

    • In-app: No obvious issue found, but there is always a keypress/tap noice when the keyboard finishes starting up. I've never heard this before. Two keypresses/taps are heard in-app and one as a system keyboard. Keyman sometimes crashes (see the beginning of the recording below.)

      crash.and.keypress.noice.-.480p.mov

    Note also that the "Get Started" page doesn't show up even though it's on in the settings.

    • System: no issue other than the mysterious keypress/tap noice.
  • TEST_BASELINE_IPAD: PASSED No obvious error is found.

    • In-app: Unlike on iPhone, the "Get Started" page is always shown at start up. I've noticed that one keypress/tap noice is heard before the page is shown and another after the page is closed.

      Screen.Recording.2021-12-14.at.2.14.25.PM.mov
    • System: no issue other than the mysterious keypress/tap noice.

@mcdurdin
Copy link
Copy Markdown
Member Author

  • Make sure no script errors are encountered

To check for script errors when testing KeymanWeb, open the Developer Console (F12 on Chrome/Windows) -- script errors are shown in red and can be located by clicking the
image button in the top right of the Developer Console.

@mcdurdin
Copy link
Copy Markdown
Member Author

This video no longer appears available?

@mcdurdin
Copy link
Copy Markdown
Member Author

  • [TEST_BASELINE_ANDROID] In-app: longpress keys are not working in simulator, but OK in a physical device.

This may not be related to this change; can we do a screenshare test together on this?

@keymanapp-test-bot retest SUITE_BASELINE TEST_BASELINE_ANDROID

@MakaraSok
Copy link
Copy Markdown

This video no longer appears available?

Screen_Recording_20211214-155720_Keyman.mp4

@MakaraSok
Copy link
Copy Markdown

MakaraSok commented Dec 20, 2021

SUITE_BASELINE: Test that nothing has changed in existing keyboards

  • TEST_BASELINE_ANDROID: FAILED??? In Keyman for Android, on an Android device or emulator, run through the above tests, both system and in-app keyboards.
  1. Run through basic usage tests of a variety of keyboards, including sil_euro_latin and khmer_angkor.

OK, but longpress doesn't work on emulator. Nothing like this is seen on a physical device.

  1. Test longpress keys, layer switching, flick-up gesture.

OK

  1. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

The key preview may disable during a speed tapping. See the screenshot recording in the comment right above.

@MakaraSok
Copy link
Copy Markdown

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

  • TEST_CAPS_ANDROID: FAILED??
  1. Run through a basic usage test of caps_lock_layer_3620.zip.

One of the basic usage of the keyboard doesn't work as expected. Long press keys are not responsive; the subkeys do not pop up as expected on Android 10 simulator API 29. Note that this doesn't happen on the physical device.

  1. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK.

  1. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK, but those keys with longpresses on the simulator. On the physical device however, this doesn't happen.

  1. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

Each key is working OK, but those keys with longpresses. On the physical device however, this doesn't happen.

  1. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

The test is not quite practical, but it does appear that the preview went off and on amid typing.

Screen_Recording_20211220-104706_Keyman.mp4

@MakaraSok
Copy link
Copy Markdown

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

  • TEST_CAPS_IPHONE: FAILED
  1. Run through a basic usage test of caps_lock_layer_3620.zip.

Nothing weird observed.

  1. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK, but the weird thing is the double tap seem unnatural to the touch. You don't hear the two taps, just one is heard. :/

  1. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK including longpresses.

  1. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

Not only that the double tap is unnatural to the touch, it doesn't enable caps layer at all. Double tap the Shift key and start typing SS gives you Ss instead.

  1. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

No physical device within reach right now.

@MakaraSok
Copy link
Copy Markdown

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

  • TEST_CAPS_IPAD: FAILED
  1. Run through a basic usage test of caps_lock_layer_3620.zip.

Nothing weird observed.

  1. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK, but the weird thing is the double tap seem unnatural to the touch. You don't hear the two taps, just one is heard. :/

  1. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK including longpresses.

  1. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

Not only that the double tap is unnatural to the touch, it doesn't enable caps layer at all. Double tap the Shift key and start typing SS gives you Ss instead.

Screen.Recording.2021-12-20.at.1.20.41.PM.mov
  1. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

No physical device within reach right now.

@MakaraSok
Copy link
Copy Markdown

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

  • TEST_CAPS_WEB: PASSED
  1. Run through a basic usage test of caps_lock_layer_3620.zip.

Nothing weird observed.

  1. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK.

  1. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK including longpresses.

  1. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

OK

  1. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

No physical device within reach right now.

@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Dec 20, 2021
@mcdurdin mcdurdin removed this from the A15S20 milestone Jan 3, 2022
@mcdurdin mcdurdin modified the milestones: A15S21, A15S22 Jan 3, 2022
Base automatically changed from feat/developer/3620-developer-ide-support to master January 17, 2022 01:13
@mcdurdin
Copy link
Copy Markdown
Member Author

Failing test feedback:

As I have now captured the three failure cases as separate issues, I will go ahead and merge this PR, with the caveat that these issues must be resolved before release.

@mcdurdin mcdurdin merged commit 2227040 into master Jan 17, 2022
@mcdurdin mcdurdin deleted the feat/web/3620-caps-layer branch January 17, 2022 19:19
@keyman-server
Copy link
Copy Markdown
Collaborator

Changes in this pull request will be available for download in Keyman version 15.0.180-alpha

@MayuraVerma
Copy link
Copy Markdown
Contributor

How to enable CAPS LOCK with double tap in Keyboard?

@MayuraVerma
Copy link
Copy Markdown
Contributor

I have tried in iOS 15.0.188-alpha with EUROSIL
no sign of caps lock with double tap on shift key.

@MayuraVerma
Copy link
Copy Markdown
Contributor

I tested full_caps_3620_3621.zip with iOS 15.0.188-alpha

CAPS lock turns off, after a key stroke
CAPS lock should stay in that state until turned off
if not there is not difference between CAPS lock and shift layer.

@mcdurdin
Copy link
Copy Markdown
Member Author

@MakaraSok or @bharanidharanj, are you able to replicate the issues that @MayuraVerma has experienced? Can you work with Mayura to build a reproducible test case so we can resolve this before release?

@mcdurdin
Copy link
Copy Markdown
Member Author

I have tried in iOS 15.0.188-alpha with EUROSIL

The SIL Eurolatin keyboard has not yet been updated to support Caps Lock 😄

@MayuraVerma
Copy link
Copy Markdown
Contributor

I have tried in iOS 15.0.188-alpha with EUROSIL

The SIL Eurolatin keyboard has not yet been updated to support Caps Lock 😄

Yes, I realized it later. since then I tested full_caps_3620_3621.zip with iOS 15.0.188-alpha

Capslock works. But I am thinking we are leaving it to the keyboard developer to choose the next layer after output in CAPS layer, default should be it should remain in CAPS layer. Once we enter CAPS layer, only why to get out of CAPS layer should be release the CAPS lock. This is how it works in hardware, we should mimic the same in software keyboard also.

@MayuraVerma
Copy link
Copy Markdown
Contributor

I have tried in iOS 15.0.188-alpha with EUROSIL

The SIL Eurolatin keyboard has not yet been updated to support Caps Lock 😄

My earlier assumption was, web/ios/android picks up the NCAPS state from kmn. looks like we need to rewrite it mobile version.

@mcdurdin
Copy link
Copy Markdown
Member Author

My earlier assumption was, web/ios/android picks up the NCAPS state from kmn.

Yeah, unfortunately given how the touch and physical keyboards diverge, we could not find a safe way to do this automatically. My preference is always to give the control to the keyboard designer, in any case -- but the trade-off is that we don't get this kind of behaviour "for free".

@mcdurdin
Copy link
Copy Markdown
Member Author

mcdurdin commented Feb 13, 2022

Once we enter CAPS layer, only why to get out of CAPS layer should be release the CAPS lock.

There are two ways, in our design, that the Caps Lock layer is released on touch:

  1. When the user touches the Caps Lock key again (or another modifier key accessible on the Caps Lock layer).
  2. When the user switches apps or input contexts. This matches the behaviour of the system keyboards -- Caps Lock state is not maintained long-term, unlike on a hardware keyboard.

So if you are seeing behaviour that does not match those conditions, that is definitely a bug. And if you can give repro steps, that will help us squash the bug!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spec: Caps Lock layer for touch layouts

5 participants