Let’s try to isolate the failure in #15565 (comment)

@bors-servo try

⌛ Trying commit 9325b29 with merge 88cc53b...

💔 Test failed - windows-gnu-dev

Yup, same:

bash -l ./mach test-unit
[…]
   Compiling layout_tests v0.0.1 (file:///C:/buildbot/slave/windows-gnu-dev/build/tests/unit/layout)
    Finished dev [unoptimized + debuginfo] target(s) in 51.84 secs
     Running C:\buildbot\slave\windows-gnu-dev\build\target\debug\deps\gfx_tests-07dc130213a68c25.exe
error: test failed
program finished with exit code 127

Let’s see with only rust updated, not cargo.

@bors-servo try

⌛ Trying commit fe5a797 with merge 6302e4c...

💔 Test failed - mac-rel-wpt2

I killed some useless jobs, here is windows-gnu-dev.

It didn't break, so the culprit is cargo, not rustc.

rust-lang/cargo@385e243...5db6d64

@bors-servo try

⌛ Trying commit f160408 with merge 0cd8a9e...

💔 Test failed - windows-msvc-dev

Killed all jobs but windows-gnu-dev.

@bors-servo try

⌛ Trying commit 7c460d8 with merge bbfe851...

💔 Test failed - mac-dev-unit

windows-gnu-dev

Works with rust-lang/cargo@ce9ddf3, so the culprit is rust-lang/cargo#3651.

The change in rust-lang/cargo#3651 was related to LD_LIBRARY_PATH handling when running executables (the PATH variable on Windows). Previously Cargo would automatically put all paths printed out by build scripts into LD_LIBRARY_PATH but this caused many bugs with system libraries were duplicated. Typically if a system directory is expected by LD_LIBRARY_PATH it's configured ahead of time (before Cargo is called), so that change is only augmenting LD_LIBRARY_PATH with paths in the build dir.

Do servo tests rely on a DLL in a system directory to be in PATH, when the dll isn't already in PATH? It'd be helpful to drill into which DLL is missing, although unfortunately I don't know of a great way to do that. I've used objdump -p foo.exe | grep -i dll in the past, but if it doesn't reproduce locally that may be difficult to simulate on the bots.

Thanks for the quick reply @alexcrichton.

I pushed the hash of the culprit PR in case someone wants to reproduce it locally (just fetch that PR and try to build it).

The required DLLs of that test (at least locally) appear to be:

        DLL Name: ADVAPI32.dll
        DLL Name: GDI32.dll
        DLL Name: KERNEL32.dll
        DLL Name: msvcrt.dll
        DLL Name: SHELL32.dll
        DLL Name: USER32.dll
        DLL Name: USERENV.dll
        DLL Name: WS2_32.dll
        DLL Name: libstdc++-6.dll
        DLL Name: LIBEAY32.dll
        DLL Name: SSLEAY32.dll

Most of those are pretty standard so the DLLs except for the last three. The libstdc++ dll is typically located in C:\msys64\mingw64\bin which I see is in PATH. The last two, libeay32 and ssleay32 are OpenSSL dlls, and at least locally they're also located in C:\msys64\mingw64\bin.

That then brings up the question of why is this failing? Or is it not failing due to dll issues?

So debugging this a little more locally it looks like the wrong libstdc++-6.dll is being loaded. Presumably mach (I think?) is placing the .servo/... rustc bin dir in PATH, and because that's a MinGW compiler it contains libstdc++-6.dll next to it. This one is quite old though and different from the one you're actually linking to, C:\msys64\mingw64\bin\libstdc++-6.dll.

So previously Cargo would insert C:\msys64\mingw64\bin into PATH at the beginning but it is now not currently doing so (assuming that the user has configured that). The two crates that print out this path, libz and openssl-sys, ironically don't actually link to libstdc++ at all, so it was just being picked up by accident!

To fix this some possible solutions could be:

• Don't use MinGW host compilers, only MSVC ones. These don't ship with any odd DLLs and you shouldn't run into any conflicts. Note that you'll want to install the MinGW target though which will allow you to still compile libraries (also the MSVC target so you can run plugins).
• Change the location of rustc in PATH or make sure that dir doesn't show up in PATH at all (the .servo downloaded compiler). That'll make sure the bad dll isn't accidentally picked up.

That's unfortunately all I can think of :(

We can probably change mach to not use PATH at all. Instead, run Cargo with an absolute path to its executable, and give it RUSTC and RUSTDOC environment variables with similar absolute paths.

@alexcrichton Doesn't that mean that the MinGW host compiler is useless now? Is @retep998 correct when qualifying this as a breaking change?

@SimonSapin I believe that would work, yeah.

@nox perhaps, I personally find cross-compiling from MSVC to MinGW far more useful, but there are many opinions here. I do not have an opinion on @retep998's comment.

(I’ve edited this PR to be only about Cargo and filed #15911 separately for the compiler update.)

I'm not 100% sure (it's been 10 years or so), but one thing that we might also be able to do is use the file element in the application manifest cargo.exe.manifest to point to the relative path where the installed file is:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa374191%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

IIRC, this comes before PATH entries in the LoadLibrary search order.

Don't use MinGW host compilers, only MSVC ones.

How hard would it be to do that @larsbergstrom?

@larsbergstrom oh that'd work perfectly if we could do that, in that case it'd basically be the same thing as rpath we use for other platforms. Mind filing a rust-lang/rust issue with that info?

@larsbergstrom Did you file an issue in the end?

@nox I did not manage to track down the details on how that stuff worked yet (it's on my backlog!). But, @metajack turned off mingw builds so you should be unblocked.