eta: 2023-02
Contributes to #1512

Quoting @mxinden "An unstable Testground is a useless Testground."
We have a few reports of Testground flakiness .

Creating this issue to gather all flakiness reports in a single location.

• This issue needs a refresh to start from the stability dashboard and then fix the flakiness.

Phase 0: Gather instances of flakiness

Ideally, we'd organize based on:

• how frequent it happens (often, sometimes, rarely),
• where it happens (macOS, Linux, arm, amd64)
• the platform (docker, docker + ci, later k8s)
• when it happens (build, setup, run)

I believe we should hold until the EKS cluster is ready to evaluate flakiness with the k8s cluster.
The method, tools, data, and metrics we came up with here will be reused for k8s.

List of cases where we've seen flakiness:

• Testground reports failure for instances when they appear to succeed #1382 (comment)
• docker build (sometimes?) times out #1397
• Flaky rust-sdk example #1400

Phase 1: Measure how flaky Testground is

• Define how we measure this flakiness
  • Likely: have a few test cases that we run in CI and on developers' machines and gather results. See Testground Interop with Stability In Mind doc below for examples of scripts.
• Define how much flakiness is acceptable
  • Examples:
    • On CI, with fresh runners, we accept: 1 / 100 failures for tests x and y
    • On a developer machine, with long-running instances, we accept: 10 / 100 failures for test x and y
    • Note that we might have to differentiate between macOS and Linux machines, given that Docker on macOS appears less stable by default.

Phase 2: Improve

Once phase 1 is done we should have a few test cases / instances of testground flakiness

Notes

• Testground Interop with Stability In Mind

Follow-up tasks:

• Tooling for EKS testing.