Part of #18782
Started at #18780

The problem is two-fold:

• requiring the pytest-astropy metapackage instead of the corresponding, direct requirements, means we don't have direct control over the lower bounds of our direct dependencies.
• pytest-astropy currently requires a superset of our actual test requirements. Notably, this means we still pull pytest-openfiles from PyPI on every single test run, even though we don't use it, and it's explicitly marked as unsupported.