MAINT: Remove astropy.test(), should use pytest directly by MridulS · Pull Request #16208 · astropy/astropy

MridulS · 2024-03-17T20:51:56Z

Blocked by:

Developer documentation overhaul #16561

This PR addresses one of the points brought forward in #16177.

This removes astropy.test(), this does not deprecate/remove TestRunner. We can do a deprecation too (astropy.test()) if folks think this is a tad bit aggressive for downstream code 😅 . We can go down the sunpy route too and add a direct call to pytest.main from astropy.test but I would argue let's keep the testing separate from final user install. A user shouldn't really ever worry about installing a broken astropy wheel. It should be caught in the CI first! If a user is doing a source build install I think it's fair to assume they are comfortable with calling pytest directly from command line.

This is the first step towards removing TestRunner. There is a lot of code in TestRunner that hasn't been touched in 10 years and things will only keep on evolving in pytest to play maintenance catchup 😅

github-actions · 2024-03-17T20:52:24Z

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

Do the proposed changes actually accomplish desired goals?
Do the proposed changes follow the Astropy coding guidelines?
Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

pllim · 2024-03-18T16:59:31Z

Thanks!

I think this still needs a change log, because we had advertised astropy.test() in RC testing in the past, for that one person who wonders why 6.1rc testing is different...

pllim · 2024-03-18T17:00:16Z

p.s. When requested changes are addressed, this also would need a "Extra CI" label.

saimn · 2024-03-18T18:18:07Z

So this would imply that RC testing should be done with pytest --pyargs astropy ?

MridulS · 2024-03-20T22:31:33Z

So this would imply that RC testing should be done with pytest --pyargs astropy ?

Yes, assuming all the paths are fine, this should work just fine. I tried this out with https://github.com/astropy/astropy/wiki/v6.0-RC-Testing and it worked fine with pytest --pyargs astropy --remote-data

mhvk

A few small further comments...

mhvk · 2024-03-25T17:44:34Z

-``astropy.test()`` to prevent test runs from taking too long. These tests can
-be run by ``astropy.test()`` by adding the ``remote_data='any'`` flag.  Turn on
+``pytest`` to prevent test runs from taking too long. These tests can
+be run by ``pytest`` by adding the ``remote_data='any'`` flag.  Turn on


This is doubled up. This line and l.308 should be combined into

be included by running ``pytest --remote-data=any``.

Done in #16561.

mhvk · 2024-03-25T17:47:12Z

-
-     import astropy
-     astropy.test(package='time')
+   changes do not break existing code, by running the Astropy tests.


This is a bit out of date in general, since we now suggest to use tox. I'd either skip the example and go straight to For more details on... or, perhaps better, rewrite as,

changes do not break existing code, by running the Astropy tests, e.g., by running:: tox -e test For more details on running ...

I am making alll the Sphinx doc changes now in #16561. For this bit I am just linking to the testing guidelines page as the one and only-one reference on running tests.

mhvk · 2024-03-25T18:53:01Z


-The easiest way to test if your installed version of ``astropy`` is running
-correctly is to use the :ref:`astropy.test()` function::
+To run tests after installing ``astropy`` we need to install ``pytest`` and


Here, the changes are good as is, since we've not installed from source.

These changes conflict with #16555, so I would suggest reverting all changes to install.rst here.

hamogu · 2024-06-12T15:10:44Z

There seems to be broad agreement by reviewers that this is the right direction to take except for the small comments by @mhvk.
@MridulS : Do you have time to add those or do you want me to add it to your PR?

MridulS · 2024-06-12T16:27:17Z

@MridulS : Do you have time to add those or do you want me to add it to your PR?

I can do it later this week, unless someone gets to it before that :)

taldcroft · 2024-06-14T14:52:09Z

This is almost certain to conflict with #16561 since that renames development_workflow.rst. It sounds like everyone is agreed on the idea of removing astropy.test(), so I think the easiest is if I apply these doc changes in #16561. This is independent of removing the code that implements astropy.test(). Sound OK?

This might imply that we don't merge this PR before #16561 so we don't have documentation of something that doesn't work.

taldcroft · 2024-06-19T16:10:59Z

-
-     import astropy
-     astropy.test(package='time')
+   changes do not break existing code, by running the Astropy tests.


I am making alll the Sphinx doc changes now in #16561. For this bit I am just linking to the testing guidelines page as the one and only-one reference on running tests.

taldcroft · 2024-06-19T16:11:25Z

-``astropy.test()`` to prevent test runs from taking too long. These tests can
-be run by ``astropy.test()`` by adding the ``remote_data='any'`` flag.  Turn on
+``pytest`` to prevent test runs from taking too long. These tests can
+be run by ``pytest`` by adding the ``remote_data='any'`` flag.  Turn on


Done in #16561.

taldcroft · 2024-06-19T16:11:43Z


    pytest astropy/wcs/index.rst

-.. _astropy.test():


Done in #16561.

taldcroft · 2024-06-19T16:12:00Z

 Pass ``-n auto`` to create the same number of processes as cores
 on your machine.

-Similarly, this feature can be invoked from ``astropy.test``::


Done in #16561.

taldcroft · 2024-06-19T16:12:19Z

 running framework. For example doctests and detailed documentation on how to
 write them, see the full :mod:`doctest` documentation.

-.. note::


Done in #16561.

taldcroft · 2024-06-19T16:15:49Z


-The easiest way to test if your installed version of ``astropy`` is running
-correctly is to use the :ref:`astropy.test()` function::
+To run tests after installing ``astropy`` we need to install ``pytest`` and


These changes conflict with #16555, so I would suggest reverting all changes to install.rst here.

github-actions · 2024-08-21T05:42:29Z

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

astrofrog · 2024-09-05T11:25:20Z

I've rebased to remove changes to the documentation but haven't changed anything else. How do people feel about deprecation vs removal?

mhvk

Looks good modulo a small change to the comments.

On the larger question, I think a deprecation error would be good, since we have been asking people to use astropy.test() since forever... I guess it can just be a __getattr__ in the main __init__.py?

mhvk · 2024-09-05T13:41:32Z

@@ -63,8 +63,8 @@ def test_find_mod_objs():

    # just check for astropy.test ... other things might be added, so we


Should adjust the comment here to be "just check for astropy.conf"

Update: This test update (with Marten's comment applied) is now #17873 . Thanks again! I added you both as co-authors.

pllim · 2024-09-05T14:07:12Z

How do people feel about deprecation vs removal?

I vote deprecation because we have been asking people to run that for years during RC testing. Release manager for next release also need to remember to modify RC testing instructions.

saimn · 2024-09-05T16:42:39Z

I think we should deprecate TestRunner directly, as proposed in #16177. If we no more use it and barely maintain it then it's better to be clear about its future.

mhvk · 2024-09-05T17:45:29Z

Thinking about it more, I'm actually OK with just removing it, as long as people trying to use astropy.test get a clear error with instructions of what to do instead.

pllim · 2024-09-05T20:34:48Z

synphot uses the test runner, so deprecating it without much notice would be annoying. I am sure a lot of packages use it (whether they know it or not) as a result of the package-template.

neutrinoceros · 2024-10-23T17:45:49Z

It doesn't seem likely to get unstuck in time for the 7.0 feature freeze so I'll remove the milestone.

github-actions · 2025-02-03T05:36:28Z

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

pllim · 2025-02-06T23:12:36Z

Direct removal without deprecation is too controversial, so I am closing this PR but I made a note on #16177 to refer back to this when we get to it in the future. Additionally, I credited Mridul as co-author over at #17732 for the change in tox.ini that is not controversial and can be merged much sooner. Thank you!

github-actions Bot added Docs installation labels Mar 17, 2024

MridulS force-pushed the remove_test_runner branch 2 times, most recently from 0facccd to b1451d0 Compare March 17, 2024 21:55

MridulS mentioned this pull request Mar 17, 2024

MAINT: Deprecate TestRunner(Base) for downstream users #16209

Closed

astrofrog requested changes Mar 17, 2024

View reviewed changes

Comment thread tox.ini Outdated

pllim added this to the v6.1.0 milestone Mar 18, 2024

pllim added Refactoring API change PRs and issues that change an existing API, possibly requiring a deprecation period testing labels Mar 18, 2024

saimn reviewed Mar 18, 2024

View reviewed changes

Comment thread astropy/utils/tests/test_introspection.py

saimn reviewed Mar 18, 2024

View reviewed changes

Comment thread docs/install.rst

MridulS force-pushed the remove_test_runner branch from becae4a to 1d01377 Compare March 20, 2024 22:20

MridulS force-pushed the remove_test_runner branch from 1d01377 to e43ba80 Compare March 24, 2024 03:29

MridulS requested review from astrofrog, mhvk and saimn March 24, 2024 03:30

mhvk reviewed Mar 25, 2024

View reviewed changes

astrofrog modified the milestones: v6.1.0, v7.0.0 Apr 4, 2024

This was referenced Apr 17, 2024

Remove photutils.test() astropy/photutils#1725

Merged

Remove regions.test() astropy/regions#549

Merged

hamogu mentioned this pull request Jun 14, 2024

Developer documentation overhaul #16561

Merged

1 task

pllim marked this pull request as draft June 14, 2024 20:14

taldcroft reviewed Jun 19, 2024

View reviewed changes

github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Aug 21, 2024

MAINT: Remove astropy.test(), should use pytest directly

3c8c238

astrofrog force-pushed the remove_test_runner branch from e43ba80 to 3c8c238 Compare September 5, 2024 11:24

astrofrog removed their request for review September 5, 2024 11:31

mhvk reviewed Sep 5, 2024

View reviewed changes

pllim removed the Close? Tell stale bot that this issue/PR is stale label Sep 5, 2024

neutrinoceros removed this from the v7.0.0 milestone Oct 23, 2024

github-actions Bot added the Close? Tell stale bot that this issue/PR is stale label Feb 3, 2025

mhvk mentioned this pull request Feb 6, 2025

TST: Do not use astropy.test() in double run #17732

Merged

1 task

pllim mentioned this pull request Feb 6, 2025

Deprecate astropy test runner in core and downstream #16177

Closed

pllim closed this Feb 6, 2025

This was referenced Mar 11, 2025

TST: Replace test with conf in test_find_mod_objs() #17873

Merged

RFC: Introduce pending deprecation warning of test runner #17874

Merged

pllim mentioned this pull request Jul 4, 2025

Remove deprecated tests/command.py module and really deprecate test runner for v8.0 #17883

Merged

3 tasks

		@@ -63,8 +63,8 @@ def test_find_mod_objs():

		# just check for astropy.test ... other things might be added, so we

Uh oh!

Uh oh!

Conversation

MridulS commented Mar 17, 2024 • edited by pllim Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 17, 2024

Uh oh!

Uh oh!

pllim commented Mar 18, 2024

Uh oh!

pllim commented Mar 18, 2024

Uh oh!

Uh oh!

Uh oh!

saimn commented Mar 18, 2024

Uh oh!

MridulS commented Mar 20, 2024

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hamogu commented Jun 12, 2024

Uh oh!

MridulS commented Jun 12, 2024

Uh oh!

taldcroft commented Jun 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 21, 2024

Uh oh!

astrofrog commented Sep 5, 2024

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pllim Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pllim commented Sep 5, 2024

Uh oh!

saimn commented Sep 5, 2024

Uh oh!

mhvk commented Sep 5, 2024

Uh oh!

pllim commented Sep 5, 2024

Uh oh!

neutrinoceros commented Oct 23, 2024

Uh oh!

github-actions Bot commented Feb 3, 2025

Uh oh!

pllim commented Feb 6, 2025

Uh oh!

Reviewers

MridulS commented Mar 17, 2024 •

edited by pllim

Loading

pllim Mar 11, 2025 •

edited

Loading