Skip to content

TST: Do not use astropy.test() in double run#17732

Merged
pllim merged 1 commit into
astropy:mainfrom
pllim:tst-double-trouble
Feb 7, 2025
Merged

TST: Do not use astropy.test() in double run#17732
pllim merged 1 commit into
astropy:mainfrom
pllim:tst-double-trouble

Conversation

@pllim

@pllim pllim commented Feb 6, 2025

Copy link
Copy Markdown
Member

Description

Had some problem with warnings being in consistent in #17570 . And given #16177 , let's see if this works.

Update: Copied incantation from #16208 but without the controversial stuff.

Blocks

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions

github-actions Bot commented Feb 6, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@github-actions

github-actions Bot commented Feb 6, 2025

Copy link
Copy Markdown
Contributor

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

@pllim pllim marked this pull request as ready for review February 6, 2025 19:44
@pllim pllim requested a review from astrofrog February 6, 2025 19:44
@pllim pllim mentioned this pull request Feb 6, 2025
1 task
@pllim

pllim commented Feb 6, 2025

Copy link
Copy Markdown
Member Author

My tox-fu is weak. Is this equivalent as running astropy.test() twice? It is green and I can see two different test logs in the double job. 🤷‍♀️

@mhvk

mhvk commented Feb 6, 2025

Copy link
Copy Markdown
Contributor

We have to be sure the tests are run twice without invoking pytest twice, and not just every test two times, but run everything once, then everything another time. I could have sworn we had the solution for that in one of the comments quite a while ago...

@mhvk

mhvk commented Feb 6, 2025

Copy link
Copy Markdown
Contributor

OK, found it: it is in #16208, you seem to need,

    double: pytest --keep-duplicates --pyargs astropy astropy

@astrofrog

Copy link
Copy Markdown
Member

Yes it's important the tests are run twice in the same interpreter

@saimn saimn modified the milestones: v7.0.1, v7.0.2 Feb 6, 2025
Co-authored-by:  Mridul Seth <mail@mriduls.com>
@pllim pllim force-pushed the tst-double-trouble branch from 4a896dc to dc8519a Compare February 6, 2025 23:06
@pllim

pllim commented Feb 6, 2025

Copy link
Copy Markdown
Member Author

Oh, thanks! I forgot about that PR. I copied Mridul's incantation over and credited him as a co-author.

@pllim

pllim commented Feb 6, 2025

Copy link
Copy Markdown
Member Author

I think it works? But hard to tell because there is only one log of the run instead of two logs as before, but the number of tests look like they doubled.

  • Normal job (minimal deps): 25271 passed, 3802 skipped, 219 xfailed in 290.69s (0:04:50)
  • Double run: 49170 passed, 7605 skipped, 430 xfailed in 387.82s (0:06:27)

@pllim

pllim commented Feb 6, 2025

Copy link
Copy Markdown
Member Author

I don't see ../../docs stuff tested in the double run though, unlike the normal job. Is this expected? Is it a problem?

Comment thread tox.ini
cov-!double: pytest --pyargs astropy {toxinidir}/docs {env:MPLFLAGS} --cov astropy --cov-config={toxinidir}/pyproject.toml --cov-report xml:{toxinidir}/coverage.xml {posargs}

double: python -c 'import sys; from astropy import test; test(); sys.exit(test())'
double: pytest --keep-duplicates --pyargs astropy astropy

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I have to do something like this?

Suggested change
double: pytest --keep-duplicates --pyargs astropy astropy
double: pytest --keep-duplicates --pyargs astropy astropy {toxinidir}/docs {toxinidir}/docs {env:MPLFLAGS} {posargs}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think so, and probably do astropy {doxinidir}/docs astropy {toxinidir}/docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can skip {env:MPLFLAGS} since, if I understand correctly, it'll always be empty in this context.
Also: the current implementation for double tests doesn't include doctests for docs/. We could add it later but that seems out of scope here.

Otherwise: neat ! I had no idea --keep-duplicates existed or that --pyargs

  • could be used with more than one package name
  • still falls back to interpreting arguments as fs-paths when they don't match an installed package name

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

astropy.test() not able to collect docs was a bug, not a feature. So if we can address that here also, would be nice.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for the other inputs, I just want to be consistent with single run commands.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think we need posargs...

- name: Python 3.11 Double test (Run tests twice)
linux: py311-test-double
runs-on: ubuntu-24.04-arm
posargs: -n=4

Anyways trying it out at #17743

@mhvk

mhvk commented Feb 7, 2025

Copy link
Copy Markdown
Contributor

I think it works? But hard to tell because there is only one log of the run instead of two logs as before, but the number of tests look like they doubled.

* Normal job (minimal deps): `25271 passed, 3802 skipped, 219 xfailed in 290.69s (0:04:50)`

* Double run: `49170 passed, 7605 skipped, 430 xfailed in 387.82s (0:06:27)`

If you look at the log, you see it has the normal ordering and then, after astropy/wcs, it goes back to astropy/c* - so, yes, this works!

@neutrinoceros neutrinoceros left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like the correct, simple solution. Very cool !
The one critical (and theoretically hard to check) question to answer for this to work should be "does it fail where the old implementation did ?".
... I'm happy to report that double test is apparently broken on my machine (yay !) so I was able to check that I get the exact same tests to fail with both invocations1. I'm now pretty confident they're indeed equivalent.

I'll approve as is as I don't think suggested additions are needed !
Thank you !

Footnotes

  1. Naturally, I intend to report (and possibly fix) them, though I'm short on time now so it'll have to wait for Monday.

Comment thread tox.ini
cov-!double: pytest --pyargs astropy {toxinidir}/docs {env:MPLFLAGS} --cov astropy --cov-config={toxinidir}/pyproject.toml --cov-report xml:{toxinidir}/coverage.xml {posargs}

double: python -c 'import sys; from astropy import test; test(); sys.exit(test())'
double: pytest --keep-duplicates --pyargs astropy astropy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can skip {env:MPLFLAGS} since, if I understand correctly, it'll always be empty in this context.
Also: the current implementation for double tests doesn't include doctests for docs/. We could add it later but that seems out of scope here.

Otherwise: neat ! I had no idea --keep-duplicates existed or that --pyargs

  • could be used with more than one package name
  • still falls back to interpreting arguments as fs-paths when they don't match an installed package name

@pllim

pllim commented Feb 7, 2025

Copy link
Copy Markdown
Member Author

Well, since Clement did prove that the change as-is is exactly equivalent replacement, maybe I can merge this first as-is. And then open follow-up PR to expand it. Thanks, all!

@pllim pllim merged commit c86b581 into astropy:main Feb 7, 2025
@pllim pllim deleted the tst-double-trouble branch February 7, 2025 15:59
meeseeksmachine pushed a commit to meeseeksmachine/astropy that referenced this pull request Feb 7, 2025
pllim added a commit that referenced this pull request Feb 7, 2025
…732-on-v7.0.x

Backport PR #17732 on branch v7.0.x (TST: Do not use astropy.test() in double run)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants