User Test Results

Test specification and instructions

User tests are not required

I can't really give much of an opinion on the validity of the tests... I've spent a couple of hours reviewing this and there's nothing glaringly obvious. Pretty concerned about the mathematical complexity of gestures and really wondering if there is a simpler path.

I've wondered as well. I've looked back at https://www.npmjs.com/package/@use-gesture/vanilla a few times, since it seems to be a pretty prominent gesture library... but if it actually is able to do what we need, the amount of time it'd take me to process the library and figure out how to do everything we want seems to be pretty high. It seems specially tuned for user-interactive UI elements & animations, rather than keyboard-oriented gestures - it's all about "move", "drag", "zoom", "scroll", and "pinch". The paradigm is different enough that it'd take a lot of code 'wrangling', as I haven't seen a simple way to 'chain' gestures while looking at its API. (Longpresses after a roaming-touch would be very tricky, let alone without 'chaining' them from component gestures... and even then, there's longpress -> subkey selection, which I don't see being modeled clearly from its paradigm either.)

I also found "zingtouch", but the demo doesn't appear to support multi-touch. Trying to start a second drag in the "swipe" demo instantly killed the first drag without appearing to model the second drag. That's not a great sign in its favor, at least not for our use case.

use-gesture, at least, doesn't die instantly when a second touch starts; it faithfully sticks with the initial touchpoint without so much as a hiccup.

I've wondered as well.

Glad to hear that you've been looking at other libraries. I wonder if some of their models of tackling gesture recognition may be helpful even if the libraries themselves don't meet our needs? Have you dug into the sources of any of them? (I'm just curious, really!)

.... I wonder if some of their models of tackling gesture recognition may be helpful even if the libraries themselves don't meet our needs? Have you dug into the sources of any of them? (I'm just curious, really!)

I hadn't, but it wasn't too hard to find use-gesture's core recognition engine code.

A few notes from a quick look:

• Main source folder of interest: https://github.com/pmndrs/use-gesture/tree/main/packages/core/src/engines
• They (naturally) expect to handle only TouchEvents - we'd have to map mouse events to them
  • ...as a fallback? There are parts that reference the PointerEvent type instead. (And I can interact with their demos via mouse, which corroborates this.)
  • PointerEvent requires Chrome 55 though, so relying solely on that is out for us, for now, b/c Android 5.0 uses older if not updated.
    • It would definitely simplify the event tracking system if we could use it, though.
• There's literally no touchpath segmentation. Considering the gestures offered... this checks out. All the gesture types they offer are simple, single-motion (per touch-point, b/c pinch) gestures.
  • When you don't need to worry about one gesture 'turning into' another gesture, things are simpler.

When your paradigm doesn't expect touchpath segmentation... all the calculations automatically assume that they're part of one continuous motion. That assumption greatly simplifies everything for them, comparatively speaking. Were we to try adopting their code, we'd have to handle segmentation ourselves when and where needed and have gestures cancelling other gestures at a relatively high level, rather than handled by the gesture-recognition engine itself, which is what our design will be aiming to do.

All of that said, pinch may make a useful reference for if and when we decide to implement caret panning. I still haven't fully wrapped my head around handling of multi-touchpoint gesture types, so that'd definitely be a helpful reference.

gestures cancelling other gestures

Remind me which gestures need cancellation by other gestures in our model?

gestures cancelling other gestures

Remind me which gestures need cancellation by other gestures in our model?

• Longpresses cancel flick processing, for one.
  • If a key's been held long enough, a flick-like motion to one of the displayed subkeys should not produce a flick gesture!
• "Roaming touches" are cancelled once any other gesture type is recognized.
• Once longpress is sufficiently long, the part caring about "length" of the press needs to drop, "lock in" the key that is longpressed, and then allow subkey selection based on that key.
  • Longpresses are multi-stage gestures.
• If in multitap mode, any recognition of longpresses or flick motion should auto-cancel the ongoing multitap state.

To name a few off the top of my head.

Longpresses cancel flick processing, for one.

• If a key's been held long enough, a flick-like motion to one of the displayed subkeys should not produce a flick gesture!

As soon as the longpress timer resolves, it puts us into "longpress" state -- the longpress subkey menu is visible. Flicks and any other gestures should be ignored while in longpress state -- all interactions are with the longpress menu. We return to base state once the user releases their finger.

• "Roaming touches" are cancelled once any other gesture type is recognized.

I'd be comfortable with dropping roaming touches altogether.

• Once longpress is sufficiently long, the part caring about "length" of the press needs to drop, "lock in" the key that is longpressed, and then allow subkey selection based on that key.

Is this the same as above?

• Longpresses are multi-stage gestures.
• If in multitap mode, any recognition of longpresses or flick motion should auto-cancel the ongoing multitap state.

I have a sense that we don't need a generic solution for resolving multi-stage gestures. If we instead can have simple tests at each point in the state machine to determine which other gestures will cancel the existing state, we may be able to simplify the entire model somewhat? It'd be good to review this together, perhaps next week? (I wish we could do this in-person, it's so much easier to whiteboard etc)

in the state machine

Reminder: without touchpath segmentation, we found it pretty difficult to develop a unified state machine - the thing that would merit use of the word "the" there. Individual, per-gesture state machines of some sort would likely be possible without segmentation, though.

I have a sense that we don't need a generic solution for resolving multi-stage gestures. If we instead can have simple tests at each point in the state machine to determine which other gestures will cancel the existing state, we may be able to simplify the entire model somewhat?

As a reminder, that sentiment is exactly what led to this - to tackling touchpath segmentation. (Especially when considering that + the notion of a singular state machine.) The resulting segments will greatly facilitate development of such "simple tests", and are that simplification for the rest of the model.

I'm not saying that there aren't other approaches we could find that wouldn't also provide simplification, though.

A few other notes:

• If we should ever desire to handle multi-segment flicks, well, we'd need some sort of 'multi-segment' detection.

  I know our current plans call for only ever handling single-segment ones, true. It's just something to consider if we think this might ever end up on our roadmap.

• Similarly re: swipe, should we ever want to tackle that for keyboards with paired lexical models, once we enable auto-correct.