A benchmark test case for measuring scroll smoothness #61998

CareF · 2020-07-22T02:07:51Z

Description

We've noticed that even when all frames are built and rasterized fast, we can still get janky scrolling (#19222).

This test is an attempt to catch such behavior, especially due to mismatch of the input frequency and frame refresh rate, which there's an attempt to fix it by #60558

The metric we use to measure the janky behavior here is the absolute value of acceleration (2nd derivative) rather than jerk as suggested in go/tq-smooth-scrolling because the input is completely linear and the expected acceleration should be strictly zero, and observed acceleration is jumping from positive to negative within adjacent frame, meaning mathematically the discrete 3-rd derivative (f[3] - 3*f[2] + 3*f[1] - f[0]) is not a good approximation of jerk (3-rd derivative) Discrete 2nd derivative (f[2] - 2*f[1] + f[0]) on the other hand is a good measure of how the scrolling deviate away from linear, and given the acceleration should average to zero within two frames, it's also a good approximation for jerk in terms of physics.

Related Issues

#19222 and its links.

Tests

This PR is itself a test.

Local test on a Moto G4 gives sample result (90Hz input):

{
  "janky_count": 506.0,
  "average_abs_jerk": 2.0151334129828427
}

The 2nd diff of the scroller's offset vs timestamp is:

where if the input event frequency is 60Hz

{
  "janky_count": 176.0,
  "average_abs_jerk": 1.4801657785671627
}

The 2nd diff of the scroller's offset vs timestamp is:

Those peaks may come from delayed frame or misalignment of 60Hz input and approximately 60Hz (~59.94Hz) refresh rate.

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I signed the CLA.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I updated/added relevant documentation (doc comments with ///).
All existing and new tests are passing.
The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
I am willing to follow-up on review comments in a timely manner.

Breaking Change

Did any tests fail when you ran them? Please read Handling breaking changes.

No, no existing tests failed, so this is not a breaking change.

flutter-dashboard · 2020-08-18T16:22:12Z

This pull request has been changed to a draft. The currently pending flutter-gold status will not be able to resolve until a new commit is pushed or the change is marked ready for review again.

packages/flutter_test/lib/src/binding.dart

CareF · 2020-08-25T18:38:01Z

The smoothness (resampler) integration test is ready for review. @liyuqian @dreveman

liyuqian

I'm curious what janky_count and average_abs_jerk are with resampleEnabled = true in your local test? I'm expecting them to be close to 0 and thus much smaller than the 60hz-input-59.94hz-display case of average_abs_jerk = 176, average_abs_jerk = 1.48 if our default resamplingOffset was chosen properly.

dev/benchmarks/complex_layout/test/scroll_smooth_measure.dart

liyuqian · 2020-08-27T04:58:47Z

dev/benchmarks/complex_layout/test/scroll_smooth_measure.dart

The integer milliseconds observed from devices look suspicious to me... Did we lose the precision somewhere in the engine? Even with 60hz input, 16ms or 17ms seem to be sufficiently different from 16.67ms which could make resampling be ~4% off?

My very early work of recording gestures shows that, all touch screen event has timestamp of milliseconds precision. For mouse input in desktop and web Flutter the event timestamp is microsecond. I think you can still find those datas in our 1:1 docs.
I think that's from the OS level, not the engine.

My apology! I tested it again on iOS and also android devices and lost that pattern. I can't find why in the beginning of my project I found that pattern. I'm removing to integer millisecond part.

dev/benchmarks/complex_layout/test/scroll_smooth_measure.dart

CareF · 2020-08-27T14:43:20Z

Local test result on a MOTO G4 is:

Task result:
{
  "success": true,
  "data": {
    "janky_count_with_resampler": 12.0,
    "average_abs_jerk_with_resampler": 0.060136749120333385,
    "droped_frame_count_with_resampler": 54,
    "janky_count_without_resampler": 426.0,
    "average_abs_jerk_without_resampler": 2.0538881567851037,
    "droped_frame_count_without_resampler": 50
  },
  "benchmarkScoreKeys": [
    "janky_count_with_resampler",
    "average_abs_jerk_with_resampler",
    "droped_frame_count_with_resampler",
    "janky_count_without_resampler",
    "average_abs_jerk_without_resampler",
    "droped_frame_count_without_resampler"
  ]
}

liyuqian

My very early work of recording gestures shows that, all touch screen event has timestamp of milliseconds precision. For mouse input in desktop and web Flutter the event timestamp is microsecond. I think you can still find those datas in our 1:1 docs.
I think that's from the OS level, not the engine.

Is that an Android specific issue or does iOS also have this issue? (I can't tell from the 1:1 doc.) I would be surprised if Fuchsia is only providing input events with millisecond-precision timestamps. CC @dreveman

dev/benchmarks/complex_layout/test/scroll_smooth_measure.dart

liyuqian · 2020-08-28T00:15:01Z

dev/benchmarks/complex_layout/android/settings.gradle

Are the following lines in the gradle file needed to make the test run, or are they just added to keep this gradle file up to date with other gradle files generated by flutter create with a newer Flutter version?

If these are specific to this test, it might be nice to document why they're needed as it doesn't look to be obvious.

It's copied from what's in flutter create and is necessary to load e2e plugins. Without this I will get:

Plugin project :e2e not found. Please update settings.gradle. /Users/minglyu/flutter/dev/benchmarks/complex_layout/android/app/src/main/java/io/flutter/plugins/GeneratedPluginRegistrant.java:4: error: package dev.flutter.plugins.e2e does not exist import dev.flutter.plugins.e2e.E2EPlugin; ^ /Users/minglyu/flutter/dev/benchmarks/complex_layout/android/app/src/main/java/io/flutter/plugins/GeneratedPluginRegistrant.java:14: error: cannot find symbol E2EPlugin.registerWith(registry.registrarFor("dev.flutter.plugins.e2e.E2EPlugin")); ^ symbol: variable E2EPlugin location: class GeneratedPluginRegistrant 2 errors FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':app:compileDebugJavaWithJavac'. > Compilation failed; see the compiler error output for details. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 27s

I didn't add comments because it's just updating this to newer version, and newer test cases in dev/benchmarks are already the newer version.

liyuqian · 2020-08-28T00:16:35Z

dev/benchmarks/complex_layout/test/scroll_smooth_measure.dart

How's this 40E3 picked?

I tried 35 and 45 and find the result qualitatively same. I'm expecting it to be a few frame interval. With single frame like 16 or 17ms the result has significant fluctuation in my early test.

dev/benchmarks/complex_layout/test/scroll_smooth_measure.dart

CareF · 2020-08-28T15:18:36Z

Adding 59Hz input, the test run locally gives:

Task result:
{
  "success": true,
  "data": {
    "janky_count_with_resampler_90Hz": 12.0,
    "average_abs_jerk_with_resampler_90Hz": 0.06177293766739098,
    "dropped_frame_count_with_resampler_90Hz": 48,
    "janky_count_with_resampler_59Hz": 21.0,
    "average_abs_jerk_with_resampler_59Hz": 0.08973802433087295,
    "dropped_frame_count_with_resampler_59Hz": 33,
    "janky_count_without_resampler_90Hz": 386.0,
    "average_abs_jerk_without_resampler_90Hz": 1.7382774686832683,
    "dropped_frame_count_without_resampler_90Hz": 74,
    "janky_count_without_resampler_59Hz": 372.0,
    "average_abs_jerk_without_resampler_59Hz": 2.563500633832529,
    "dropped_frame_count_without_resampler_59Hz": 23
  },
  "benchmarkScoreKeys": [
    "janky_count_with_resampler_90Hz",
    "average_abs_jerk_with_resampler_90Hz",
    "dropped_frame_count_with_resampler_90Hz",
    "janky_count_with_resampler_59Hz",
    "average_abs_jerk_with_resampler_59Hz",
    "dropped_frame_count_with_resampler_59Hz",
    "janky_count_without_resampler_90Hz",
    "average_abs_jerk_without_resampler_90Hz",
    "dropped_frame_count_without_resampler_90Hz",
    "janky_count_without_resampler_59Hz",
    "average_abs_jerk_without_resampler_59Hz",
    "dropped_frame_count_without_resampler_59Hz"
  ]
}

CareF · 2020-08-28T15:43:39Z

My very early work of recording gestures shows that, all touch screen event has timestamp of milliseconds precision. For mouse input in desktop and web Flutter the event timestamp is microsecond. I think you can still find those datas in our 1:1 docs.
I think that's from the OS level, not the engine.

Is that an Android specific issue or does iOS also have this issue? (I can't tell from the 1:1 doc.) I would be surprised if Fuchsia is only providing input events with millisecond-precision timestamps. CC @dreveman

My apology! I tested it again on iOS and also android devices and lost that pattern. I can't find why in the beginning of my project I found that pattern. I'm removing to integer millisecond part. @liyuqian
But luckily I didn't see significant difference after the change, or actually during one version of my previous commit it's already not important.

liyuqian

LGTM

CareF · 2020-08-29T14:45:11Z

All other ios based driver tests failed. Now my google account is deactivated, I can't see the test log any more but it's possible the failure is related to the introduction of dev_dependencies on e2e. @liyuqian PTAL, something I did to android/settings.gradle may be also necessary to ios/...

This reverts commit 12b7355.

…" (#64883) This reverts commit 12b7355.

liyuqian · 2020-08-31T18:41:40Z

Thanks for the quick fix in #64885! I'll keep an eye on it when it lands. Meanwhile, I created #64964 as I believe the test logs should be made available to public.

…64990) This reverts commit df83398.

…r#61998)" (flutter#64883) This reverts commit 12b7355.

…tter#64885)" (flutter#64990) This reverts commit df83398.

CareF self-assigned this Jul 22, 2020

fluttergithubbot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Jul 22, 2020

googlebot added the cla: yes label Jul 22, 2020

CareF marked this pull request as draft July 22, 2020 02:07

fluttergithubbot added the work in progress; do not review label Jul 22, 2020

CareF changed the title ~~A benchmark for measuring scroll smoothness~~ A benchmark test case for measuring scroll smoothness Jul 22, 2020

CareF mentioned this pull request Aug 3, 2020

Resample input events due to irregular events delivery or VSYNC and input frequency mismatch #41118

Closed

CareF commented Aug 25, 2020

View reviewed changes

packages/flutter_test/lib/src/binding.dart Outdated Show resolved Hide resolved

CareF removed the work in progress; do not review label Aug 25, 2020

CareF marked this pull request as ready for review August 25, 2020 18:37

flutter-dashboard bot added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Aug 25, 2020

CareF requested review from dreveman and liyuqian August 25, 2020 18:37

liyuqian reviewed Aug 27, 2020

View reviewed changes

CareF requested a review from liyuqian August 27, 2020 15:06

liyuqian reviewed Aug 28, 2020

View reviewed changes

CareF added 10 commits August 28, 2020 08:19

add smoothness measure test

8b62aaa

update metrics

73c7929

Formatting and licence header

70c6d1a

update pubspec checksum

6a0e91f

use pointerdata

1ea35da

use test variant

110775c

add to device lab

fa5c17b

modify according to liyuqian@

ef03614

typo fix

ca0b4f6

scroll_smooth_measure.dart => measure_scroll_smoothness.dart

adae105

CareF added 2 commits August 28, 2020 08:29

update comments

9fc27ee

update comments and variable names

651830a

add 59Hz input

3b96d48

remove millisecond integer

465beab

CareF requested a review from liyuqian August 28, 2020 16:45

liyuqian approved these changes Aug 29, 2020

View reviewed changes

CareF added the waiting for tree to go green label Aug 29, 2020

fluttergithubbot merged commit 12b7355 into flutter:master Aug 29, 2020

CareF deleted the smooth_scrolling_bench branch August 29, 2020 14:35

CareF restored the smooth_scrolling_bench branch August 29, 2020 14:45

shihaohong pushed a commit that referenced this pull request Aug 30, 2020

Revert "A benchmark test case for measuring scroll smoothness (#61998)"

95adf90

This reverts commit 12b7355.

shihaohong mentioned this pull request Aug 30, 2020

Revert "A benchmark test case for measuring scroll smoothness" #64883

Merged

shihaohong pushed a commit that referenced this pull request Aug 30, 2020

Revert "A benchmark test case for measuring scroll smoothness (#61998)…

34541c3

…" (#64883) This reverts commit 12b7355.

liyuqian mentioned this pull request Aug 31, 2020

Devicelab logs should be available for public #64964

Closed

fluttergithubbot pushed a commit that referenced this pull request Sep 1, 2020

Reland #61998 and update complex_layout ios files (#64885)

df83398

jonahwilliams pushed a commit that referenced this pull request Sep 1, 2020

Revert "Reland #61998 and update complex_layout ios files (#64885)" (#…

28de49e

…64990) This reverts commit df83398.

mingwandroid pushed a commit to mingwandroid/flutter that referenced this pull request Sep 6, 2020

A benchmark test case for measuring scroll smoothness (flutter#61998)

9133fab

mingwandroid pushed a commit to mingwandroid/flutter that referenced this pull request Sep 6, 2020

Revert "A benchmark test case for measuring scroll smoothness (flutte…

874f6c0

…r#61998)" (flutter#64883) This reverts commit 12b7355.

mingwandroid pushed a commit to mingwandroid/flutter that referenced this pull request Sep 6, 2020

Reland flutter#61998 and update complex_layout ios files (flutter#64885)

a605c4b

mingwandroid pushed a commit to mingwandroid/flutter that referenced this pull request Sep 6, 2020

Revert "Reland flutter#61998 and update complex_layout ios files (flu…

619fddd

…tter#64885)" (flutter#64990) This reverts commit df83398.

liyuqian mentioned this pull request Sep 11, 2020

Reland "perf test for measuring scroll smoothness" #65602

Merged

liyuqian mentioned this pull request Sep 24, 2020

complex_layout_android__scroll_smoothness generates unexpected numbers #66608

Closed

CareF mentioned this pull request Dec 17, 2020

complex_layout_android__scroll_smoothness janky_count_without_resampler_90Hz regression #72450

Closed

github-actions bot locked as resolved and limited conversation to collaborators Aug 31, 2021

A benchmark test case for measuring scroll smoothness #61998

A benchmark test case for measuring scroll smoothness #61998

Uh oh!

Conversation

CareF commented Jul 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Tests

Checklist

Breaking Change

Uh oh!

flutter-dashboard bot commented Aug 18, 2020

Uh oh!

Uh oh!

CareF commented Aug 25, 2020

Uh oh!

liyuqian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liyuqian Aug 27, 2020

Choose a reason for hiding this comment

Uh oh!

CareF Aug 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CareF Aug 28, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CareF commented Aug 27, 2020

Uh oh!

liyuqian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

liyuqian Aug 28, 2020

Choose a reason for hiding this comment

Uh oh!

CareF Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liyuqian Aug 28, 2020

Choose a reason for hiding this comment

Uh oh!

CareF Aug 28, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CareF commented Aug 28, 2020

Uh oh!

CareF commented Aug 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liyuqian left a comment

Choose a reason for hiding this comment

Uh oh!

CareF commented Aug 29, 2020

Uh oh!

liyuqian commented Aug 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CareF commented Jul 22, 2020 •

edited

Loading

CareF Aug 27, 2020 •

edited

Loading

CareF Aug 28, 2020 •

edited

Loading

CareF commented Aug 28, 2020 •

edited

Loading