Skip to content

Disable tests using gauge#50367

Merged
dnfield merged 1 commit intoflutter:masterfrom
dnfield:disable_test
Feb 7, 2020
Merged

Disable tests using gauge#50367
dnfield merged 1 commit intoflutter:masterfrom
dnfield:disable_test

Conversation

@dnfield
Copy link
Contributor

@dnfield dnfield commented Feb 7, 2020

This test is flaking at an unacceptable rate, and is incompatible with certain versions of iOS and certain versions of Xcode that we need to be able to test on.

It is also preventing another test from detecting regresions on an iPhoneX, which is what the target was original set up for.

/cc @digiter @godofredoc @keyonghan

issue tracking this: #45435

See also: #49787

I plan to land this TBR on red to make infra less flaky.

@Hixie
Copy link
Contributor

Hixie commented Mar 17, 2020

I just closed a bug that was complaining about this test being flaky. We should probably just delete the files since nobody is championing them.

@dnfield
Copy link
Contributor Author

dnfield commented Mar 17, 2020

#45435 is the bug tracking re-enabling. @gaaclarke @liyuqian and I all looked at making it work at some point but it's not trivial and it's probably not sustainable in its current form.

@flar
Copy link
Contributor

flar commented Mar 18, 2020

The post_backdrop_filter tests were representing a concern from a developer. The CPU/GPU measurements helped us track the state of the fix that helped drastically reduce the energy usage for their app. The fact that we have no replacement for that measurement is of big concern. The energy usage could be tracked manually using Xcode and its energy graphing tools, but that can't be automated the same way that gauge was doing for us until it got broken by the upgrade to Xcode 11.

@liyuqian has more details on the developers who have been running into that problem.

The fix that was being tested for regression by these benchmarks is flutter/engine#14104

@dnfield
Copy link
Contributor Author

dnfield commented Mar 18, 2020

I don't think anyone is opposed to testing this. We just need tests that are reliable enough to not close the tree and do not require the devicelab to stay on a fixed version of iOS we can't keep it on.

@liyuqian
Copy link
Contributor

I'm pushing the work to get the measurement tool work again, and our test to be reliable and not flaky. Please stay tuned :)

@flar
Copy link
Contributor

flar commented Mar 23, 2020

Originally I thought this was just getting rid of the CPU/GPU measurements of these benchmarks, but it looks like they were deleted entirely. Note that the benchmarks measure both the standard ms/frame factors and they also separately produce the CPU/GPU numbers. We can disable their dependency on the flutter gauge mechanism independently of their other role as a general benchmark.

Was deleting them in their entirety the intended goal here, or just the dependency on gauge?

@dnfield
Copy link
Contributor Author

dnfield commented Mar 23, 2020

Just the dependency on gauge. It's fine to add them back if they don't depend on that.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants