Disclaimer: I'm not entirely sure this is the root cause of the failing tests, because I haven't been able to reproduce it in a controlled environment.

Important observations:

• The failing tests are not failing in isolation. E.g. the test MlWithSecurityIT test {yaml=ml/trained_model_cat_apis/Test cat trained models} itself reads from the index pattern ml-stats-*, but never creates it. This means in isolation it will always find no results (no matching indices), and the test succeeds.
• Therefore, an ml-stats-000001 must be lingering from a previous test. However, all tests start with wiping Elasticsearch, including all indices (see: ESRestTestCase#wipeCluster).

I think the following is the root cause of the failing tests:

• TrainedModelStatsService collecting inference stats and writing them to Elasticsearch via a scheduler that triggers every second.

This leads to the following failing sequence of events:

• Some test executes an infer trained model call, and schedules writing stats, which will happen between 0 and 1 secs.
• Test finishes.
• New test starts, and wipes the cluster including the ML stats index.
• Stats writing triggers, which triggers creating the ML stats index; during index creation, the index is temporarily in a state during which reading from it fails.
• Concurrently, however, the new test reads from this index, leading to the "all shards failed" failure.

The following should fix it (in theory):

• MachineLearning#cleanUpFeature clearing the trained model stats queue.
• Note that before each test, the feature reset API is called before deleting all indices, which is the correct order for this fix to work.

Fixes: #62699 #121726 #123034 #123200 #124168 #125641 #125642 #134257 #125750 #125909 #126299 #127625 #131364 #133440