ci: fix flaky binstall-check job#2170
Conversation
Two failure modes were causing red CI on unrelated PRs and on every release-plz PR: - Unauthenticated api.github.com requests hit 403 under contention. Pass GITHUB_TOKEN so binstall's GitHub API calls are authenticated. - Release PRs bump the in-tree version to one that is not published yet, so '--manifest-path .' made binstall look for a non-existent release. Drop the flag so binstall resolves the latest published version, and skip the job entirely on release-plz branches.
Dropping it caused binstall to resolve the latest published release (0.24.1), whose archive still has the pre-#2165 layout, so the job fails with 'lychee bin not found'. Using the in-tree manifest is the whole point: it validates the metadata before the next release.
There was a problem hiding this comment.
Thanks for fixing :)
Yeah, manifest path is needed but it's still an imperfect test. It's installing the latest release archive (which is likely older than master) using the binstall metadata from master.
I don't know if there's a good way around this.. maybe we can try to install a nightly release archive? It would still be 24 hours behind, but it would let us test with the upcoming release binaries.
I think this is fine. We basically want to lint/test if binstall can parse our metadata. The archive contents are not really that relevant. I found out that it's possible to specify In other words, I don't think we would detect any more errors if we test against nightly instead of latest release with Going to merge now. If there actually still is an improvement to be made it can be done in a fallow up PR. |
|
With testing nightly, the purpose would be to guard against changes in the release archive structure and to catch this before the official release is pushed. But it's not super urgent and there's probably not many more release changes coming up. |
|
With v0.24.2 the job fails :/ but maybe rightfully so? Edit: huh, seems to be fixed after rerunning the job 🤷 |
|
It might be using the version from Cargo.toml trying to install the new version before the binaries are uploaded. Just a guess though. |
The binstall-check job from #2165 fails for two reasons:
Should unblock #2169 and future release-plz PRs.