steps will still have a :failed status, and that will cause test-bot
to exit with an error.

Which ones? I'd rather those steps were also skipped rather than us ignoring failures.

The ones we skipped. Doing skipped doesn't make the :failed status go away; it just adds the formula to the @skipped_or_failed_formulae array.

Here's an example: https://github.com/Homebrew/homebrew-core/runs/4115979271?check_suite_focus=true#step:7:177

Doing skipped doesn't make the :failed status go away

Yeh, I don't think it should. I think it should still show up as "FAILED". If the issue is the exit code: can we handle that instead?

Hmmm, we could. Let me have a look.

I haven't been able to work out how to handle the exit code without tweaking the @status a bit, but the changes I'm about to push should still show FAILED in log when something fails, because of

and

The next time @status is read is when determining the exit code, so it seems that that's the only sensible way to handle that.

I believe the exit code is determined here:

which comes from

so I don't currently see a sensible way of changing the exit code without changing some steps to not become :failed. I've tried to minimise the steps I'm doing this for to the minimum required. I'll also add some output at the end to indicate that some failures were ignored.

I've updated this to handle build/linkage/test failures for unbottled dependents too.

The idea is to downgrade all these errors into warnings that will still display failure messages but won't cause test-bot to exit with an error. I'm thinking it might be helpful to get @BrewTestBot to post comments on a PR about warnings in the CI log to indicate that they should be looked at, since no one will read the logs otherwise when CI is green.

I'm thinking it might be helpful to get @BrewTestBot to post comments on a PR about warnings in the CI log to indicate that they should be looked at, since no one will read the logs otherwise when CI is green.

I believe Jenkins used to do this and it made the issue overview so much easier.

I was wondering why @BrewTestBot stopped doing that. We could maybe roll our own, but there's the issue of access to secrets on PR branches.

warnings in the CI log to indicate that they should be looked at, since no one will read the logs otherwise when CI is green.

Would GitHub Actions annotations help? Like the CI coverage annotations we have in Homebrew/brew.

They would help, yes. Easier to miss than comments, but better than hidden away behind a green CI log.

Yeh, I think something like that would be ideal for adding warnings 👍🏻