Hi @sharwell ,
Let us start with this.

1. I haven't actually executed this locally. Please let me know tips and tricks about local runs.
2. I suppose this should be implemented some other way for WpfTheories or we should exclude them.
3. I do not know if returning the time from the first attempt is enough or if we should send more to the message bus. See point 1.

Just a thought, but I suggested once that we create a separate leg for integration tests that we considered flaky. That way they still ran and we could monitor progress on moving them to the "stable" leg. I think I would prefer a solution more in that vain than simply rerunning failed tests.

@JoeRobich we're going to run some experiments in this branch and come back with clear data before attempting to make a decision.

Edit: I'll keep a running list of our experimental results here. Items marked "Pass with retry" are cases that would have failed without this pull request, but instead passed.

Total completed legs in the experiment: 24
Total pass rate for legs in the experiment: 92%
Pass rate without the retry feature: 75%

One of the two failures that occurred with retry will be resolved by #36190.

This LGTM, just waiting on @jaredpar to approve expanding the experiment by merging this.