Skip to content

Import standard library types at runtime#18191

Merged
pllim merged 1 commit intoastropy:mainfrom
eerovaher:runtime-type-imports
Jun 5, 2025
Merged

Import standard library types at runtime#18191
pllim merged 1 commit intoastropy:mainfrom
eerovaher:runtime-type-imports

Conversation

@eerovaher
Copy link
Member

@eerovaher eerovaher commented May 29, 2025

Description

So far imports that are needed only for type checking have been placed inside if TYPE_CHECKING: blocks to avoid importing them at runtime. However, because of bugs in Sphinx runtime imports should be preferred.

This pull request should be backported to solve some documentation build problems downstream packages might have (such as astropy/astroquery#3327).

Approvals by sub-package

Sub-packages that have not been approved by 2025.06.12 will be updated in separate pull requests.

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions
Copy link
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

Copy link
Member

@nstarman nstarman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably won't fully resolve Sphinx-caused issues, but it's bound to help!
And it will also help with runtime type checkers which is an added bonus

Edit: let me eat my words. Sphinx has many problems with resolving type annotations.
Edit: I think the error here is in Python, not Sphinx! types.EllipsisType isn't explicitly documented in the types module so intersphinx isn't seeing it.

import urllib.request
import zlib

url = "https://docs.python.org/3/objects.inv"
with urllib.request.urlopen(url) as f:
    inv = f.read()

print(b"types.EllipsisType" in inv)
# False

What are our options? And what solution might automatically propagate downstream?

@github-project-automation github-project-automation bot moved this from Triage to Reviewer approved in Cosmology, the Expansion May 29, 2025
@mhvk
Copy link
Contributor

mhvk commented May 30, 2025

I see this goes back to sphinx-doc/sphinx#9813 which is commented as being tricky to solve. The solution here seems just partial, though: if it were so easy, wouldn't sphinx just set TYPE_SETTING = True?

@pllim
Copy link
Member

pllim commented May 30, 2025

Why were they in if to begin with? Any cons we need to know about unleashing this as mandatory runtime import?

@nstarman
Copy link
Member

nstarman commented May 30, 2025

@pllim it was driven by 2 things: a ruff config to prefer to put type-time imports in a type-checking block and the thought that it could prevent increased import times. The former has come under flack for the difficulty it causes runtime type checkers (and sphinx). The latter can be true, but doesn't appear to be noticeable in most (any?) cases thus far.

@mhvk
Copy link
Contributor

mhvk commented May 30, 2025

Thanks, @eerovaher and @nstarman, interesting to see that the whole if TYPE_CHECKING thing is getting a bit of a second thought. Of course, the import loop problem can be seriously annoying, and it is hard to see how to solve that for annotations other than by something like putting it in a if TYPE_CHECKING block...

@nstarman
Copy link
Member

nstarman commented May 30, 2025

Stdlib and 3rd party shouldn't be problematic, but yeah 1st party is interesting.
Re. First party annotations, the suggested recourse is

  1. Full paths
    Instead of
if TYPE_CHECKING:
    from mylib import y

def func() -> y: ...

Do

if TYPE_CHECKING:
    import mylib

def func() -> mylib.y: ...
  1. Stringification (works, never preferred) but even for this opt 1 helps with runtime type checkers.

  2. Wait for that new Python PEP for forward refs. But if 1 and 2 work in the interim we should do those. 3 will eventually fix most things.

So far imports that are needed only for type checking have been placed
inside `if TYPE_CHECKING:` blocks to avoid importing them at runtime.
However, because of bugs in Sphinx runtime imports should be preferred.
@eerovaher eerovaher force-pushed the runtime-type-imports branch from eb78e51 to ac60737 Compare June 4, 2025 14:28
@eerovaher
Copy link
Member Author

I already had to fix a merge conflict and the longer this stays open, the more likely that I will have to fix more. To prevent this from getting stuck in review I am setting a deadline for next Thursday (2025.06.12). If this is not merged by that time then I will limit this pull request to the sub-packages with approvals and start opening separate pull requests for the remaining sub-packages, one sub-package per pull request.

@nstarman, did you approve only for cosmology or also for units?

Copy link
Member

@taldcroft taldcroft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for io, utils. I also accidentally reviewed all the io/builtin/ files and they seemed fine.

@nstarman
Copy link
Member

nstarman commented Jun 5, 2025

@eerovaher. Both.

@pllim
Copy link
Member

pllim commented Jun 5, 2025

Basically all subpackages have the same kind of fixes, so if we decide that runtime import isn't a problem, this is probably good to go. Any outstanding discussions left?

@pllim pllim added the benchmark Run benchmarks for a PR label Jun 5, 2025
@eerovaher
Copy link
Member Author

Runtime imports from the standard library should not be controversial and this pull request is not concerned with third party or first party imports.

Copy link
Member

@pllim pllim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this does fix downstream doc build, and there are 2 approvals, already, let's get this in. If more downstream packages (besides astroquery and specutils) report a problem, then at least we can just do a bugfix release then.

Thanks!

p.s. If you can think of a good way to avoid future typing stuff from breaking downstream before merge, please share your ideas. Even with #17861 , it does not check doc build. Having a CI to build all downstream doc from a PR branch here would be unexplored territory.

@pllim pllim merged commit 09795ab into astropy:main Jun 5, 2025
34 checks passed
@github-project-automation github-project-automation bot moved this from Reviewer approved to Done in Cosmology, the Expansion Jun 5, 2025
@lumberbot-app
Copy link

lumberbot-app bot commented Jun 5, 2025

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout v7.1.x
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 09795abdae6ebf71bc58b04be52faf79e198a1a2
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #18191: Import standard library types at runtime'
  1. Push to a named branch:
git push YOURFORK v7.1.x:auto-backport-of-pr-18191-on-v7.1.x
  1. Create a PR against branch v7.1.x, I would have named this PR:

"Backport PR #18191 on branch v7.1.x (Import standard library types at runtime)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

@pllim

This comment was marked as resolved.

@nstarman
Copy link
Member

nstarman commented Jun 5, 2025

p.s. If you can think of a good way to avoid future typing stuff from breaking downstream before merge, please share your ideas.

Agreed this is a tricky problem. The issue isn't that the types aren't valid – they are - but that sphinx et al has a damnable time rendering some types under certain conditions. I'm not sure how we effectively test for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants