Context

When running XHarness workloads, we try to recognize infrastructure issues from XHarness' exit code and in some cases, we retry/reboot.

The logic for this is stored in several ps1/sh scripts here: https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner

Problem

The MonoVM team runs tests by starting an xUnit runner which then executes shell commands that are part of every unit test. This way, the shell commands cannot access the shell tools we have prepared for the user (such as retry/reboot requests). The unit tests are also unable to float the XHarness exit code back to the wrapper script as it's nested inside of the xUnit test runner.

The problem is that we now have to duplicate the logic we have in the XHarness Helix SDK scripts and share it with dotnet/runtime where it is copied to.

We should be able to have just one place where we maintain this and allow dotnet/runtime to use this.

Goal

We should probably take advantage of the diagnostics.json file that XHarness creates and when we're uploading the contents to cloud, we should analyze it as we go and request retry/reboot.

This way we will also reduce the logic from 2 .sh and 1 .ps1 file to a 1 common Python place.

It will also be easier for us to call the Helix retry/reboot as that is python code too.