feat(web): gesture touchpath segmentation core 🐵#7058
feat(web): gesture touchpath segmentation core 🐵#7058jahorton merged 23 commits intofeature-gesturesfrom
Conversation
User Test ResultsTest specification and instructions User tests are not required Test Artifacts
|
54518ad to
cf31ca1
Compare
mcdurdin
left a comment
There was a problem hiding this comment.
I think LGTM? I've added a lot of comments, but they're mostly nits. The code looks well structured and commented, but I am not up-to-speed with all the math so I cannot really comment usefully on that.
I look forward to seeing how this progresses.
| * Range: floating-point values on the interval [0, 1]. | ||
| */ | ||
| private get angleRSquared() { | ||
| // https://www.ebi.ac.uk/thornton-srv/software/PROCHECK/nmr_manual/man_cv.html may be a useful |
There was a problem hiding this comment.
Links tend to bit-rot over time. Can you provide a summary here?
There was a problem hiding this comment.
Uh... that's a bit of an ask. I'm currently unable to explain exactly why the definition is what it is, unlike with normal regression formulas and expressions - it's new to me and seems to have a more complex derivation than standard linear regression. I can at least understand enough to interpret it and utilize it, though.
Upon another look, I can at least make https://en.wikipedia.org/wiki/Directional_statistics#Distribution_of_the_mean work here instead; might take a little clarification of how things map, but the parallel is about as clear.
Granted, yeah, it's still a link, but at least Wikipedia is good at taking steps to prevent link-rot.
| * A repeating sample indicates lack of motion, which is valuable | ||
| * information for stats-based segmentation. | ||
| */ | ||
| private repeatTimer: number | NodeJS.Timeout; |
There was a problem hiding this comment.
NodeJS.Timeout -- is this going to be a problem in web?
There was a problem hiding this comment.
Not at all. In Web, it uses the number part, while in Node, it uses the NodeJS.Timeout. setTimeout's signature is different between the two.
- https://developer.mozilla.org/en-US/docs/Web/API/setTimeout
- https://nodejs.org/en/docs/guides/timers-in-node/#leaving-timeouts-behind
We don't care about the Node object's properties; all we care about is its use as a "handle" for timeout-related functions. That aspect is 100% compatible with DOM's use of a number for the "handle" instead.
There was a problem hiding this comment.
Gotcha, it's a type definition, so goes away on JS side.
| // .mean('v') < 80: the mean speed (taken timestamp to timestamp) does not exceed 80px/sec. | ||
| if(segmentation.pre.mean('v') < 80 && segmentation.post.mean('v') > 80) { | ||
| console.log("desegmentation exception"); | ||
| return; | ||
| } | ||
| if(segmentation.pre.mean('v') > 80 && segmentation.post.mean('v') < 80) { | ||
| console.log("desegmentation exception"); | ||
| return; | ||
| } |
There was a problem hiding this comment.
This value of 80 pixels/sec seems to me to be likely to be dependent on the hardware. Higher-res devices will have a greater threshold, lower-res devices a lower threshold. So not sure we can depend on this.
In any case, this is a constant that should be defined somewhere outside this function.
There was a problem hiding this comment.
This value of 80 pixels/sec seems to me to be likely to be dependent on the hardware. Higher-res devices will have a greater threshold, lower-res devices a lower threshold. So not sure we can depend on this.
I agree. I settled for this for now because...
- It at least gets us something functional that we can use to move forward.
- There's probably some underlying principle that I can infer from how well it's working that should be adaptable to different resolutions. It'd probably be helped by
devicePixelRatiostuff, for one.
It's definitely a point worth revisiting and marking as a to-do on the base feature branch should a better solution be deferred for now. (Which it probably will.)
| @@ -0,0 +1,5 @@ | |||
| namespace com.keyman.osk { | |||
| export class Segment { | |||
| // TODO: stuff. | |||
|
Just realized one stats identity I used fairly regularly might be clarified by mentioning another Wikipedia article: one of the most underlying math / stat principles used may be found at https://en.wikipedia.org/wiki/Covariance. (This is also at the heart of Look at the second equation seen on the page. The same pattern holds for variance, too - just replace Y with X again. In case it helps with the parse, This is what allows us to safely 'batch' the statistical values (and 'unbatch' them with |
Co-authored-by: Marc Durdin <marc@durdin.net>
| // Prevent overly-short intervals / over-segmentation. | ||
| if(nextCandidate.pre.duration * 1000 < this.SLIDING_WINDOW_INTERVAL / 2) { | ||
| break; | ||
| } else if(nextCandidate.post.duration * 1000 < this.SLIDING_WINDOW_INTERVAL / 2) { | ||
| break; | ||
| } |
There was a problem hiding this comment.
- Just noticed that I neglected to fix this section when I shifted
.durationfrom seconds to milliseconds.
This PR begins phase 2 of gesture-recognizer module development - adding analysis code that can segment the path taken by touchpoints (and mouse cursors) into "segments" that may be used to build & model full-fledged gestures.
The core features and functionality added here may be found in
CumulativePathStatsandPathSegmenter. It's taken a fair bit of research, trial, and error, but I'm pretty happy with how well the core segmentation algorithm & data tracking are working now. It's even showing a lot of promise toward supporting gesture types we've triaged beyond 16.0 - I've tested quite frequently with two-segment flick paths and it's handling those pretty admirably.All of that said, sometimes it's dividing in a few places I don't want it to - in particular for slightly curved, quickly-moving motions. (Segmented regression's pretty good at picking up the shift in angle and noting when the change is very consistent!) It also splits when there's a notable shift in speed without a direction change... but that can be useful for a desired feature, so I don't mind that too much. I'd also strongly prefer over-segmentation to under-segmentation, since the former's much easier to resolve through other means.
As this phase of the feature's still in development, it's not currently "publishing" to the target API/design for the touchpath structure... let alone providing clean public
Segmentclasses. So, for now, I have quite a number ofconsole.logstatements in place to provide a view into what's happening & how it's working. Those console statements will be removed in an imminent followup... this one's already quite large enough as it is, looking at the displayed diff stat, and the changes necessary to publish a clean form of the data would add even more code to this PR.A later "following" follow-up will be to add segmentation unit tests.
So, in regard to that console stuff... here's an example case of inputting a caret-shaped motion: 'ne', then 'se':
It turns out that each
'move'segment has three subsegments, and on examining one set, the reason is because of speed:Here's the results of trying a longpress -> subkey style interaction:
@keymanapp-test-bot skip
Will probably not merge without its followups b/c console logging long-term is ugly. (As in, unless otherwise instructed.)