This is what some people call "an epic" in JIRA-land.

A long, long time ago, in the same galaxy, astropy existed when Python was youngish, pytest was younger (remember when we bundled pytest as a single file?) , CI on GitHub was almost non-existent (ask Tom R about Jenkins). There was a need to be able to test installed versions of astropy. And thus, the test runner was created. For many years, it served us well.

We even advertised it in package-template (https://github.com/astropy/package-template/blob/36ed48db74c9a0bfcf214e14a1a3fb1fdaae0c6c/%7B%7B%20cookiecutter.package_name%20%7D%7D/%7B%7B%20cookiecutter.module_name%20%7D%7D/_%7B%7B%20cookiecutter._parent_project%20%7D%7D_init.py#L13), so downstream packages may have similar runners too (whether they realized it is there or not is another issue).

Now in 2024, we have multi-architecture CI and wheels. The need for such a test runner has diminished greatly. In fact, having it might even hinder progress like these:

• Do not include tests in Astropy wheels (reduce wheel size for AWS Lambda) #16152
• Move astropy to src directory #13678

Additionally, it is not aging well now that pytest is very complicated and there are plenty of pytest plugins available:

• Running tests from astropy.test() fails when asdf-astropy is installed #16165

All the while, it was never feature complete:

• import astropy;astropy.test() hangs under macOS #10667
• astropy.test() should be able to test docstrings #8153

The plea to get rid of the test runner went as far back as 2018 (#8153 (comment)), "Friends, I beg of you, let's please consider axing the test runner entirely."

Proposal

1. ✔ Open issues in affected downstream packages to warn them to check whether they inherited this test runner from package template (or added it intentionally) and tell them to move away from using it. Alternative is to invoke pytest directly (or tox, whatever).
2. Stop using test runner internally within core lib and docs (including RC testing instructions). -- RFC: Introduce pending deprecation warning of test runner #17874
3. Step 4 but pending=True -- RFC: Introduce pending deprecation warning of test runner #17874
4. Deprecate astropy.test() function and astropy.tests.runner module. Look at the diff from MAINT: Remove astropy.test(), should use pytest directly #16208 for an idea on what files to touch. -- Remove deprecated tests/command.py module and really deprecate test runner for v8.0 #17883
5. After a deprecation period, remove the deprecated stuff from astropy. If downstream packages have not caught up, they need to pin to an older release of astropy or have a broken test runner.

Impact: High!
Benefit: Less technical debt to carry forward.
Cost: Some crying maintainers.

Counter proposal

We keep everything as-is, but we have to start testing astropy.test() in our CI.