And it also solves the double compilation issue.

I don't feel qualified to understand the check we're removing and why it's not necessary.

r? @petrochenkov

@bors r+

📌 Commit 3aca663 has been approved by estebank

@bors r-

You can't remove this, UI tests should check ERROR/WARN/etc annotations.
See #46116 and linked issues for the background.

Even with the background, I don't understand its purpose. For this we typically had compile-fail tests. What's the point to check the compiler errors if you compare outputs? If an error/warning/help/whatever isn't thrown out by the compiler, the test will fail.

The other solution which I quickly started would be to actually parse the compiler stderr. And I can assure you that this is not gonna be fun or efficient...

I thought that ui tests were using the rendered field in the json output...

@GuillaumeGomez

Even with the background, I don't understand its purpose.

TLDR: UI testing alone is very unreliable.
Suppose you have a test with N lines that should report an error and M lines that should not report an error. How do you ensure this by having only the auto-generated .stderr file? "Manually" looking through two files trying to find corresponding lines is not reliable, it's very easy to miss something, it's like separating asserts from run-pass tests into separate files. ERROR/WARN annotations must be inline and also not auto-generated.

For this we typically had compile-fail tests.

The plan so far is to gradually migrate them to UI to consistently detect changes in secondary stuff like spans in addition to ERROR annotations (#44844).

@estebank

I thought that ui tests were using the rendered field in the json output...

This was the case until #48337 got merged.

@GuillaumeGomez
I see two ways to resolve #48550:

• The simpler one: Revert 1dc2015, figure out the reason why the "for more information try rustc --explain E0XXX" stuff doesn't appear in text converted from json and fix it. If the fix is somehow fundamentally impossible, then test the "explain" message in a couple of dedicated tests explicitly using --error-format=human and leave all the remaining tests in pre-Rustc explain #48337 state (this also means 1300 less lines of text in the repo, yay).
• The more complex, but maybe ultimately better one: Make // ~ERROR/// ~WARN checking work on --error-format=human output. This is what @nikomatsakis proposed originally (https://internals.rust-lang.org/t/consolidating-compile-fail-and-ui-testing-and-possibly-others-as-well/4896).

Then I'll just go for the --error-format=human output checking. I had someone working in most cases so I think it's doable.

Closing in favor of #48684