Once #44925 is finished, it will be pretty easy to add automatic re-running of individual ciGroups to add some protection and against flaky tests.
In subsequent work, we will likely be removing ciGroups altogether, in favor of running small test suites in a sort of queue-based fashion. In this case, re-running tests will mean re-running a small suite instead of a full ciGroup.
Other considerations:
- How should we make visible / notify users about flaky tests? Should that be a separate issue?
- Will we be re-running ciGroups so often (on PRs that have constantly breaking tests, for example), that it's not worth it?
Once #44925 is finished, it will be pretty easy to add automatic re-running of individual ciGroups to add some protection and against flaky tests.
In subsequent work, we will likely be removing ciGroups altogether, in favor of running small test suites in a sort of queue-based fashion. In this case, re-running tests will mean re-running a small suite instead of a full ciGroup.
Other considerations: