Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

Interesting. Looks like the --run-slow flag (and corresponding @pytest.mark.slow marker) are defined in some plugin that's pulled from pytest-astropy, but I haven't been able to identified which one yet.

of course, it's from pytest-astropy itself.

Haha yes... It also has --run-hugemem, see https://github.com/astropy/pytest-astropy/blob/main/pytest_astropy/plugin.py

So yeah I am not sure if this is feasible without refactoring pytest-astropy itself... Maybe open discussion on astropy-dev if you still want to pursue?

I do want to continue experimenting with this, but I don't think the problems I'm after can be addressed from pytest-astropy itself without breaking its users.

(and with the impeding feature freeze, I'll have to postpone this work until at least next week)

Turns out vendoring the contents of that plugin within our root conftest.py is actually trivial. I just suspected it could not be, which is why I didn't prioritize this earlier.
Any way this should be much more stable now. Note that it's still based of #18780 though, but I think I could avoid that coupling. I'll try this next once CI is clear.

I'm seeing a couple unrelated failures caused by #18880

other failures I'm seeing are already handled in existing PRs

The challenge isn't code, but people. We wanted the plugin stuff in pytest-astropy so other packages could also use it without requiring astropy, but they were so trivial that we did not want to create a whole new package just for them.

Now, they been upstream long enough that we probably have a number of packages using them, and have to keep maintaining them. So copying them over is just duplicating code.

This definitely need more discussion.

we could easily make these markers into their own plugin if duplication must be avoided, but for what it's worth:

• duplicating ~50 lines of code seems like a (much) lesser evil than irreproducible builds
• I don't think we have to make this change throughout the entire ecosystem; it's diminishing returns from here, but astropy being a core package means (to me) it needs to adhere to strict(er) quality requirements

Moving them anywhere now would create churn. We have to find and inform all the affected downstream packages. And then go through deprecation period and face some resistance. And then still keep them going while deprecated. And then finally we have to remove them.

We have to find and inform all the affected downstream packages.

why ? couldn't we just keep pytest-astropy as the "batteries included" option, even if we stop dog-fooding it in astropy itself ? I glanced over it and it didn't seem too burdensome to maintain (no release for 2 years and it's still working for everyone, as far as I can tell). Am I missing anything ?

So you are saying we create a new package for the plugin and then include that in pytest-astropy, instead of moving it here?

Not really. I would prefer that the code we use to test the core library live in the same repo, which is what the current state of the branch does, and leave pytest-astropy live its life as the provider of these functionality for everyone else that choose to adopt it.

But if we must avoid code duplication at all cost, there are other avenues. Though, it seems we agree that they are highly disruptive and undesirable.

That said, it would make sense to isolate the plugin into its own package and make pytest-astropy (and maybe astropy itself) depend on it. It shouldn't have user-facing implications, would avoid duplication entirely, and would allow anyone who needs the plugin to drop other pytest plugins from pytest-astropy they don't need.
But that can also be done in the future, if judged necessary, rather than now

As mentionned in #18882, I think my port of the slow test marker doesn't work after all, which could explain at least some of the failures seen in CI. I'll take this one back to the shop for now.

Got it. What's broken is specifically runnings test with --pyargs: since the markers are only defined in a repo-level conftest, they are not available when running the suite for an installed package. I think I can make this much easier to understand and debug using pytest's strict markers, but I'll still need to think about how to fix it. Right now, a separate, minimal plugin package looks like the only option to me.

I rebased this on top of #18890 to prevent further avoidable mistakes with marker registration, and I moved the plugin parts from pytest-astropy to our other, embedded conftest.py, hoping this will work in all conditions. If it doesn't, I'll need to upstream the discussion.

Welp, that didn't work. Let's take this outside. astropy/pytest-astropy#73

Since the markers must be provided by a pytest plugin, let's try using a patched version of pytest-astropy (from my fork) that doesn't include any dependency besides pytest.

I'll remove the "extra CI" label for now. I agree it should run when this is stable enough, but it doesn't seem worth the extra CPU time at the current proof-of-concept stage.

Can the markers be defined in astropy/conftest.py and also in docs/conftest.py? Such duplication isn't good but we are duplicating conftest.py contents already. And perhaps the markers are not needed in docs, in which case we could define them only in astropy/conftest.py.

Can the markers be defined in astropy/conftest.py and also in docs/conftest.py? Such duplication isn't good but we are duplicating conftest.py contents already

neither will be discovered by pytest when running from an arbitrary location with --pyargs so I think the marker have to be provided by a plugin.