Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ping @MikeZappa87 @aznashwan @gabriel-samfira @TBBle

@kiashok any thoughts to why this happened?

Not sure it's related, but I noticed hcsshim had similar problems with a specific MingGW version (10.2.0) and pinned to (10.3.0) to fix it: https://github.com/microsoft/hcsshim/pull/1799/files.

They also have this note:

          # Install sometimes fails when downloading mingw zip from source-forge with:
          #  "ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url"
          # Issue is with accessing from source-forge, which version 10.3+ do not use, but cannot upgrade versions.

So perhaps what's actually happened is a silent download failure and missing mingw32-make is the first symptom, like hcsshim saw. But I don't see kind of failure there.

So perhaps what's actually happened is a silent download failure and missing mingw32-make is the first symptom, like hcsshim saw.

hmm. So, it seems reasonable to pin the version.

Actually, looking at the logs of the failure supplied, and the run here on this PR, after

Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).

We see a bunch of ShimGen Lines missing in the failure case, presumably that's why executables can't be found.

https://community.chocolatey.org/packages/mingw#comment-6290804217 notes this is happening to others, and the suggestion is to uninstall the old version first. The logs show an upgrade from 8.10 and and so perhaps 8.10 used shims. I guess the new release 13.0 doesn't use shims, per this comment.

I also note hcsshim upgrade their Chocolatey install in a previous step, that might help if the left-behind shim are fixed in newer Chocoatey.

Edit: A quick look at the two Chocolatey packages versions https://community.chocolatey.org/packages/mingw and https://community.chocolatey.org/packages/mingw/12.2.0.03042023#files suggests some changes in how the package is installed so they don't get shims generated. Perhaps what's happened is that when old package is being upgraded to the new package, all the shims are left behind, and are higher in $PATH precedence than the actual files.

That's consistent with the errors (presumably coming from the shims) going looking for the real binary in a self-relative directory (where the previous version's binaries were), but 13.x installs to a different directory.

tl;dr Try uninstalling mingw first, and perhaps upgrading Chocolatey before that.

Also, hcsshim has a specific check for mingw-make at the end of their "Install mingw" step, so if something like goes wrong again, the correct test failes.

@kiashok I think that makes sense, we can merge this one in first since CI is passing. Would be great to see the recommended method in a follow up!

@kiashok I think that makes sense, we can merge this one in first since CI is passing. Would be great to see the recommended method in a follow up!

sure! I can work on the follow up PR