Conversation
|
I just closed a bug that was complaining about this test being flaky. We should probably just delete the files since nobody is championing them. |
|
#45435 is the bug tracking re-enabling. @gaaclarke @liyuqian and I all looked at making it work at some point but it's not trivial and it's probably not sustainable in its current form. |
|
The post_backdrop_filter tests were representing a concern from a developer. The CPU/GPU measurements helped us track the state of the fix that helped drastically reduce the energy usage for their app. The fact that we have no replacement for that measurement is of big concern. The energy usage could be tracked manually using Xcode and its energy graphing tools, but that can't be automated the same way that gauge was doing for us until it got broken by the upgrade to Xcode 11. @liyuqian has more details on the developers who have been running into that problem. The fix that was being tested for regression by these benchmarks is flutter/engine#14104 |
|
I don't think anyone is opposed to testing this. We just need tests that are reliable enough to not close the tree and do not require the devicelab to stay on a fixed version of iOS we can't keep it on. |
|
I'm pushing the work to get the measurement tool work again, and our test to be reliable and not flaky. Please stay tuned :) |
|
Originally I thought this was just getting rid of the CPU/GPU measurements of these benchmarks, but it looks like they were deleted entirely. Note that the benchmarks measure both the standard ms/frame factors and they also separately produce the CPU/GPU numbers. We can disable their dependency on the flutter gauge mechanism independently of their other role as a general benchmark. Was deleting them in their entirety the intended goal here, or just the dependency on gauge? |
|
Just the dependency on gauge. It's fine to add them back if they don't depend on that. |
This test is flaking at an unacceptable rate, and is incompatible with certain versions of iOS and certain versions of Xcode that we need to be able to test on.
It is also preventing another test from detecting regresions on an iPhoneX, which is what the target was original set up for.
/cc @digiter @godofredoc @keyonghan
issue tracking this: #45435
See also: #49787
I plan to land this TBR on red to make infra less flaky.