This allows both CodeTracking and Revise to pass their tests, thanks!

But yes, it could use a test.

Writing a test for this is a pretty big thing (need to construct a package and have it precompile and load a dependency) etc so I'll merge this now since it improves the situation and open an issue for a test.

Correction: Revise is still failing tests even with this merged, see e.g., https://ci.appveyor.com/project/timholy/revise-jl/builds/35276008/job/r1g63mu0t0l3b58x (windows) and https://travis-ci.org/github/timholy/Revise.jl/jobs/728286437 (linux). DEV-947 should be (barely) enough to include this. But for me locally test Revise works fine; I'm guessing it works for some paths but not others, and the CI machines fall in the category of "not."

Feel free to make fake PRs on Revise if you need it to debug this. Also, using PkgTemplates seems to suffer from the same error, and I still trigger that locally, so perhaps that's easier.

I think the stdlibs need to be updated at runtime because right now the build bot path gets cached.

But that looks like something else, will check it out.

I think #37625 should fix it.