Skip to content

feat(web): first gesture-recognizer host page 🐵#6878

Merged
jahorton merged 26 commits intofeat/web/gesture-recognizer-mainfrom
feat/web/gesture-recognizer-rigging
Jul 20, 2022
Merged

feat(web): first gesture-recognizer host page 🐵#6878
jahorton merged 26 commits intofeat/web/gesture-recognizer-mainfrom
feat/web/gesture-recognizer-rigging

Conversation

@jahorton
Copy link
Copy Markdown
Contributor

@jahorton jahorton commented Jun 30, 2022

Ah, the joys of CSS.

Rather than attempt to gerry-rig KMW itself to allow testing of intermediate states of the new module, a better strategy is to create a page that can host it directly. It's then all the better if we can give that page configuration options that will allow us to test the module's behavior under different conditions.

Presenting: the initial host page!

image

Each radio button may be used to select one of several configurations for the module to use for further interactions on the page. That little "legend" there corresponds to certain configuration "zones", and the page-default above directly corresponds to the standard config used by KMW for its touch OSKs.

Of course, it'd help to test this on touch devices too, right? Well, after some further CSS tweaking...

image

Don't worry, it'll layout just fine for constrained-width mobile devices. Unless you go really constrained, anyway.

Showcased above is a markedly different configuration. The gray box indicates the area corresponding to a base OSK, while the beige corresponds to a subkey menu. (If the gray part makes thing look "too busy", it's a simple change on my end to remove it - or perhaps, add a deactivation option.) Here, the blue "roaming behavior" range corresponds to a hypothetical roaming range that might be utilized to address #1025 and #1886.

Lest you think I spent too much time playing with CSS... most of the colored elements there, themselves, can be nigh-directly used as part of the module's configuration. The module will automatically use each element's current layout for input processing - so this allows easy live configuration "hotswapping", which is fantastic for demonstration purposes. (We can use the bounding boxes for the elements themselves at runtime as the boundaries / "zone" definitions for the engine.)


There is a new test page added to Web's testing index: see the "Standalone gesture recognizer" link. I also reorganized the index; it's gotten a bit cluttered, and we probably want to list the pages in an approximate order of relevancy.


User Testing

All testing should be done on the specialized Web testing page currently labeled "Stand-alone gesture recognition" - the same page seen in the screenshots above.

Each of these tests will mention a specific "configuration" (set of buttons to select) to use. To keep confusion down, the test names will generally use an identifier based on the numerical position of its configuration's enabled buttons.

I will use the words "touch" and "touchpoint" throughout the tests below. Many of these tests may be done by mouse instead; interpret them as "click" and "cursor location" as needed.

TEST_1-1-1_SIMPLE: (No device restrictions) Test with a standard "KMW in mobile browser" configuration:

  • Config:
    • Mobile phone page
    • Top-overflow only
    • On-screen keyboard
    • (Any)
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch within the blue "roaming" area, then end the ongoing touch.
  3. If any text was added to the logging area lower on the page, FAIL this test.
  4. Touch near the center of the yellowish "primary gesture receiver" area.
  5. Without releasing the touchpoint, move it into the blue "roaming" area, then back into the yellow area.
  6. Then release the touchpoint.
  7. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry should include state: end. If it does not exist, FAIL this test.
    • There should be no entry stating state: cancel. If it exists, FAIL this test.

TEST_1-1-1_SAFE_ZONE: (No device restrictions) Test with a standard "KMW in mobile browser" configuration:

  • Config:
    • Mobile phone page
    • Top-overflow only
    • On-screen keyboard
    • Generous
      • This will make the test easier to execute, but it's possible to run with any of the safe-zone options.
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch near, but on the left-hand side of the inner, more-lightly-colored, right-side vertical dashed line.
  3. Without releasing the touchpoint, move it past the outer, darker-colored, line to its right - but NOT into the thick black border.
  4. Then release the touchpoint.
  5. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry may include state: end. Do not fail this test, either way.
    • There should be at least one entry stating state: cancel. If it does not exist, FAIL this test.

TEST_1-1-1_SAFE_PROXIMITY: (No device restrictions) Test with a standard "KMW in mobile browser" configuration:

  • Config:
    • Mobile phone page
    • Top-overflow only
    • On-screen keyboard
    • Generous
      • This will make the test easier to execute, but it's possible to run with any of the safe-zone options.
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch near the center of the yellowish "primary gesture receiver" area.
  3. Without releasing the touchpoint, move it past the inner, lighter-colored, line to its right - but NOT past the darker set of dashed lines.
  4. Then release the touchpoint.
  5. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry should include state: end. If it does not exist, FAIL this test.
    • There should be no entry stating state: cancel. If it exists, FAIL this test.

TEST_2_1_1_TOP_BORDER: (No device restrictions) Test with a standard "mobile-app keyboard" configuration:

  • Config:
    • WebView-embedded emulation
    • Top-overflow only
    • On-screen keyboard
    • (Any)
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch near the center of the yellowish "primary gesture receiver" area.
  3. Without releasing the touchpoint, move it up into the blue "roaming" area, past the thick black border.
  4. Then release the touchpoint.
  5. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry may include state: end. Do not fail this test, either way.
    • There should be at least one entry stating state: cancel. If it does not exist, FAIL this test.

TEST_2_1_1_SIDE_BORDER: (No device restrictions) Test with a standard "mobile-app keyboard" configuration:

  • Config:
    • WebView-embedded emulation
    • Top-overflow only
    • On-screen keyboard
    • Generous
      • This will make the test easier to execute, but it's possible to run with any of the safe-zone options.
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch near the center of the yellowish "primary gesture receiver" area.
  3. Touch near, but on the right-hand side of the inner, more-lightly-colored, right-side vertical dashed line.
    • But on the left-hand side of the outer, darker-colored dashed line beside it.
  4. Without releasing the touchpoint, move it past the thick black border to its right.
    • This may easier to perform with a mouse.
  5. Then release the touchpoint.
  6. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry may include state: end. Do not fail this test, either way.
    • There should be at least one entry stating state: cancel. If it does not exist, FAIL this test.

TEST_2_3_2_ROAMING: (No device restrictions) Test using a app-hosted keyboard-style popup-key-like configuration:

  • Config:
    • WebView-embedded emulation
    • Expanded popup-key range
    • Subkey menu (right-aligned)
    • (Any)
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch near the center of the yellowish area.
  3. Without releasing the touchpoint, move it beneath the yellowish area, staying within the blue "roaming" area.
  4. Now move the touchpoint to the left of the yellowish area, then the top, always staying within the blue "roaming" area.
  5. Then release the touchpoint.
  6. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry should include state: end. If it does not exist, FAIL this test.
    • There should be no entry stating state: cancel. If it exists, FAIL this test.

TEST_2_3_2_CANCEL: (No device restrictions) Test using a app-hosted keyboard-style popup-key-like configuration:

  • Config:
    • WebView-embedded emulation
    • Expanded popup-key range
    • Subkey menu (right-aligned)
    • (Any)
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch near the center of the yellowish area.
  3. Without releasing the touchpoint, move it somewhere within the inner set of lightly-dashed lines that is colored white - that is neither yellow nor blue.
  4. Then release the touchpoint.
  5. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry may include state: end. Do not fail this test, either way.
    • There should be at least one entry stating state: cancel. If it does not exist, FAIL this test.

TEST_4_3_2_SAFE_ZONE: (No device restrictions) Test using a desktop-style popup-key-like configuration:

  • Config:
    • Desktop bottom-right floater
    • Expanded popup-key-range
    • Subkey menu (right-aligned)
    • Generous
      • This will make the test easier to execute, but it's possible to run with any of the safe-zone options.
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Touch within the yellowish area near, but on the left-hand side of the inner, more-lightly-colored, right-side vertical dashed line.
  3. Without releasing the touchpoint, move it past the darker dashed-line to its right.
  4. Now move the touchpoint below the yellowish area to a spot that is also left of the original, lightly-colored dashed line while within the blue "roaming" area.
  5. Then release the touchpoint.
  6. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
    • The last entry may include state: end. Do not fail this test, either way.
    • There should be at least one entry stating state: cancel. If it does not exist, FAIL this test.

TEST_3_2_1_MULTITOUCH: Required: a PHYSICAL device with a touchscreen

  • Config:
    • Desktop mid-page floater
    • Loose omni-directional
    • On-screen keyboard
    • (Any)
  1. If the logging area near the page's bottom has text, click the "Reset logs" button to clear it.
  2. Use one finger to touch within the yellowish area, then move it slightly upward.
    • This will be referred to as "touchpoint A."
  3. Without releasing the touchpoint, use another finger to touch elsewhere within the yellowish area.
    • This will be referred to as "touchpoint B."
  4. Now move touchpoint "B" beyond the top of the blue box, then release it.
  5. Then slightly move touchpoint A, staying within the yellow region, and release it.
  6. Check the logs for the following:
    • The first entry should include state: start. If it does not exist, FAIL this test.
      • Make note of its identifier value - this is for touchpoint A.
    • The last entry should include state: end. If it does not exist, FAIL this test.
      • This should have the same identifier value as touchpoint A.
    • There should be at least one entry stating state: cancel.
      • If it has an identifier matching that of touchpoint A (as noted above), FAIL this test.
      • If there is no state: cancel entry whatsoever, FAIL this test.
    • In the middle of the logs, you should be able to find a different state: end entry after all of the state: cancel entries.
      • Its identifier should not match that of touchpoint A. If it does, FAIL this test.

@jahorton jahorton added this to the A16S5 milestone Jun 30, 2022
@keymanapp-test-bot keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Jun 30, 2022
@keymanapp-test-bot
Copy link
Copy Markdown

keymanapp-test-bot bot commented Jun 30, 2022

User Test Results

Test specification and instructions

  • TEST_1-1-1_SIMPLE (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_1-1-1_SAFE_ZONE (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_1-1-1_SAFE_PROXIMITY (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_2_1_1_TOP_BORDER (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_2_1_1_SIDE_BORDER (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_2_3_2_ROAMING (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_2_3_2_CANCEL (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • TEST_4_3_2_SAFE_ZONE (PASSED) (notes)
  • TEST_3_2_1_MULTITOUCH (PASSED): Retested as per @jahorton suggestions and yes...now I am able to see the 'state:end' entry after scrolling down to the results pane in Android 11.0 Mobile device. (notes)

Test Artifacts

@mcdurdin
Copy link
Copy Markdown
Member

Of note at this time is that this page is not in any of the folders addressed by CI... nor is the corresponding build product, as KMW doesn't currently link it in. So... apologies on not having that available as a directly-accessible build artifact at this time.

Let's just put this under web/testing/gesture-recognizer. Then you'll get CI for free.

@jahorton
Copy link
Copy Markdown
Contributor Author

jahorton commented Jul 4, 2022

Of note at this time is that this page is not in any of the folders addressed by CI... nor is the corresponding build product, as KMW doesn't currently link it in. So... apologies on not having that available as a directly-accessible build artifact at this time.

Let's just put this under web/testing/gesture-recognizer. Then you'll get CI for free.

I thought about doing that... but then there's the matter of the module's build artifact. Or should I 'copy' that over to Web's build folder as part of the module's build?

@mcdurdin
Copy link
Copy Markdown
Member

mcdurdin commented Jul 4, 2022

I thought about doing that... but then there's the matter of the module's build artifact. Or should I 'copy' that over to Web's build folder as part of the module's build?

Yeah, this is an ongoing complexity of putting some of web's modules under common.

In some ways, it's redundant having some things under /common/web -- /web is shared between four platforms already (just like /core is shared between three, soon four), but only ever as a built artifact and never (currently) as independent modules... I am not sure of the cleanest way to tidy this up:

  1. Move all /common/web, /common/predictive-text and /common/models to something under /web/source, or
  2. ??? what else could we do? Updating all our infrastructure to have online tests for /common/web seems redundant and complicated too.

In any case, we don't want to do either of those things at this point -- we've done enough shuffling of folders for 16.0. So for now, copying build artifacts out to the /web build path seems like a reasonable solution.

@jahorton
Copy link
Copy Markdown
Contributor Author

jahorton commented Jul 4, 2022

I thought about doing that... but then there's the matter of the module's build artifact. Or should I 'copy' that over to Web's build folder as part of the module's build?

Yeah, this is an ongoing complexity of putting some of web's modules under common.

In some ways, it's redundant having some things under /common/web -- /web is shared between four platforms already (just like /core is shared between three, soon four), but only ever as a built artifact and never (currently) as independent modules... I am not sure of the cleanest way to tidy this up:

  1. Move all /common/web, /common/predictive-text and /common/models to something under /web/source, or

The problem there is one major proposal we have on the future roadmap - the idea of reusing Web's OSK in the desktop platforms. At that point, OSK code will need to live in common - and this new module will absolutely be a part of that code.

Then, because the OSK includes the predictive-text banner, that means, at minimum, needing access to the @keymanapp/models-types module - or something that near-duplicates a few important types there.

  1. ??? what else could we do? Updating all our infrastructure to have online tests for /common/web seems redundant and complicated too.

I know we'd rather "cross that bridge when we come to it" - and noting the quote part below, we'll still do that - but what would you expect us to do in this regard should we succeed in fully modularizing the OSK and reusing it for the desktop versions of Keyman?

In any case, we don't want to do either of those things at this point -- we've done enough shuffling of folders for 16.0. So for now, copying build artifacts out to the /web build path seems like a reasonable solution.

👍

@jahorton jahorton force-pushed the feat/web/gesture-recognizer-rigging branch from b0f6fcb to 9db2da6 Compare July 11, 2022 06:05
@jahorton
Copy link
Copy Markdown
Contributor Author

jahorton commented Jul 14, 2022

New host-page feature!

image

Due to an update for the previous PR in this chain, this page can now facilitate selection of different safe-zone modes. The "very generous" mode, in particular, may come in quite handy for safe-zone oriented user tests on touch devices.


In retrospect, it looks like I never updated the original screenshots after adding the first level of safe-zone visualization. For comparison, the old, fixed setting is now called "double-spaced":

image

The true KeymanEngine for Web defaults are quite tight to the device borders, which is probably good for real, physical devices... but not so much for emulated setups like this.

Base automatically changed from feat/web/gesture-recognizer-bounds to feat/web/gesture-recognizer-main July 15, 2022 05:35
@bharanidharanj
Copy link
Copy Markdown

  • TEST_1-1-1_SIMPLE (PASSED): Tested this as per the instructions and it is working as expected.

  • TEST_1-1-1_SAFE_ZONE (PASSED): Tested this as per the instructions and it is working as expected.

  • TEST_1-1-1_SAFE_PROXIMITY (PASSED): Tested this as per the instructions and it is working as expected.

  • TEST_2_1_1_TOP_BORDER (PASSED): Tested this as per the instructions and it is working as expected.

  • TEST_2_1_1_SIDE_BORDER (PASSED): Tested this as per the instructions and it is working as expected.

  • TEST_2_3_2_ROAMING (PASSED): Tested this as per the instructions and it is working as expected.

  • TEST_2_3_2_CANCEL (PASSED): Tested this as per the instructions and it is working as expected.

@bharanidharanj
Copy link
Copy Markdown

  • TEST_4_3_2_SAFE_ZONE (FAILED): Tested this as per the instructions and it seems that after moving it past the darker dashed-line to its right, I am seeing the entry stating that 'state: cancel'.

@bharanidharanj
Copy link
Copy Markdown

  • TEST_3_2_1_MULTITOUCH (FAILED): Tested this as per the instructions in my Android Mobile device Version 11.0 and it seems that the last entry 'state : end' was missing.

@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Jul 18, 2022
background-color: whitesmoke;
}

.safe-zone-color {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'-color' is a bit of a misnomer.

@jahorton
Copy link
Copy Markdown
Contributor Author

  • TEST_4_3_2_SAFE_ZONE (FAILED): Tested this as per the instructions and it seems that after moving it past the darker dashed-line to its right, I am seeing the entry stating that 'state: cancel'.

Huh. Wrote that one a while ago... and it looks like that one's on me; I wrote the failure conditions incorrectly. Totally should expect a "cancel" there.

TEST_4_3_2_SAFE_ZONE: PASSED

(fake)_TEST_WRITE_FAILURE_CONDITIONS_CORRECTLY: FAILED

@jahorton
Copy link
Copy Markdown
Contributor Author

jahorton commented Jul 19, 2022

  • TEST_3_2_1_MULTITOUCH (FAILED): Tested this as per the instructions in my Android Mobile device Version 11.0 and it seems that the last entry 'state : end' was missing.

@bharanidharanj
Noting the element's scrollbar, did you try scrolling the logging element to the bottom?

@keymanapp-test-bot retest TEST_3_2_1_MULTITOUCH

@keymanapp-test-bot keymanapp-test-bot bot added user-test-required User tests have not been completed and removed user-test-failed labels Jul 19, 2022
@bharanidharanj
Copy link
Copy Markdown

  • TEST_3_2_1_MULTITOUCH (PASSED): Retested as per @jahorton suggestions and yes...now I am able to see the 'state:end' entry after scrolling down to the results pane in Android 11.0 Mobile device.

@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Jul 19, 2022
@jahorton jahorton merged commit 5b9482c into feat/web/gesture-recognizer-main Jul 20, 2022
@jahorton jahorton deleted the feat/web/gesture-recognizer-rigging branch July 20, 2022 08:54
Copy link
Copy Markdown
Member

@mcdurdin mcdurdin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little late, but these comments were pending on this PR

Comment on lines +22 to +23
let screenX = coord.x - document.body.scrollLeft;
let screenY = coord.y - document.body.scrollTop;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let screenX = coord.x - document.body.scrollLeft;
let screenY = coord.y - document.body.scrollTop;
const screenX = coord.x - document.body.scrollLeft;
const screenY = coord.y - document.body.scrollTop;

@@ -0,0 +1,105 @@
<html>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<html>
<!DOCTYPE html>
<html>

@@ -0,0 +1,105 @@
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta charset="utf-8" />

@jahorton jahorton mentioned this pull request Sep 19, 2022
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants