Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

So, what do you think? It caught one failure in astropy-healpix but that has nothing to do with astropy stuff (astropy/astropy-healpix#258). Is this useful for a path towards #12889 ?

monthly CRON

I was not sure how useful a scheduled job is. The cron does not run on backport branches, so it would not help if the goal is to prevent regression is bugfix branch before release. As for main, it is way more useful for downstream to use our nightly wheel for their testing instead of the other way around. We just cannot ask them to do that for bugfix candidate because we do not generate dev wheels from bugfix branch.

The "Extra CI" and workflow_dispatch is for release managers when they want to kick off the release process. Usually they would open a PR with rendered change log, so that would be a good time to run this too. At least that is how I understood what Tom R and Stuart M requested.

test the published packages

That is done over at https://github.com/astropy/astropy-integration-testing . I didn't put that here because I don't think them not releasing soon enough should be a blocker for our bugfix release, as long as the compatibility is already fixed in their dev. They can release whenever they want.

Idea is fine but this raises some questions:

• What should be done if one of the packages fail for an unrelated reason, during (or just before) the release process ? Also if it stops after the first failure we won't know if another package is broken before fixing the previous one... (as here with healpix).
• Can't we already do that with astropy-integration-testing if we use the nightly wheels ?

What should be done if one of the packages fail for an unrelated reason, during (or just before) the release process?

🤷‍♀️ I would defer to @astrofrog and @Cadair who requested this feature.

Also if it stops after the first failure

My tox-fu is weak. If there is a way to break those packages up into different jobs without polluting our regular CI settings, please let me know. I want all this stuff to be contained within [testenv:downstream] section only.

Can't we already do that with astropy-integration-testing if we use the nightly wheels ?

Only for astropy nightly wheel from main. We do not provide nightly wheel from backport branch. The astropy-integration-testing could theoretically pull in backport branch but @astrofrog and @Cadair want this to run here so bugfix release can be fully automated.

My tox-fu is weak. If there is a way to break those packages up into different jobs without polluting our regular CI settings, please let me know. I want all this stuff to be contained within [testenv:downstream] section only.

Not sure. Maybe an option would be a shell script that run all commands, store their exit codes and combine them.

Only for astropy nightly wheel from main. We do not provide nightly wheel from backport branch.

Right, but all changes in bugfix branch are also in main, so seems unlikely we break bugfix without breaking main ?

so seems unlikely we break bugfix without breaking main ?

The problem is opposite, i.e., we intentionally break compatibility in main but accidentally backported it.

Hmm maybe @mhvk has a point about testing release downstream instead of dev downstream... what you all think?

From a purist's perspective, it seems to me that this doesn't belong in astropy itself. I'm not opposed to having this sort of jobs somewhere within the org, and I agree it could be our collective responsibility to run and maintain them, but it really looks to me like something that belongs in astropy-integration-testing, even if that means we cannot use tox for this specific task.

To fully automate bugfix release, these checks has to tie into publish workflow to ensure a bugfix does not break downstream.

I think it should still be possible to have the workflow be defined in another repo and be used here, but I'm not experienced enough with workflow reusability to say this confidently.

Not urgent, so I re-milestoned it. Probably no point to backport at this point.

Looks like response is lukewarm maybe cold in general, so I am closing this without merge.