Skip to content

Workaround Python3.9 limitations in test_jit_py3#51088

Closed
malfet wants to merge 1 commit intomasterfrom
malfet/make-test_jit_py3-py3.9-compatible
Closed

Workaround Python3.9 limitations in test_jit_py3#51088
malfet wants to merge 1 commit intomasterfrom
malfet/make-test_jit_py3-py3.9-compatible

Conversation

@malfet
Copy link
Copy Markdown
Contributor

@malfet malfet commented Jan 26, 2021

In Python-3.9 and above inspect.getsource of a local class does not work if it was marked as default, see https://bugs.python.org/issue42666 #49617
Workaround by defining make_global function that programmatically accomplishes the same

Partially addresses issue raised in #49617

@facebook-github-bot facebook-github-bot added cla signed oncall: jit Add this issue/PR to JIT oncall triage queue labels Jan 26, 2021
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jan 26, 2021

💊 CI failures summary and remediations

As of commit ff8869c (more details on the Dr. CI page):


💚 💚 Looks good so far! There are no failures yet. 💚 💚


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Copy link
Copy Markdown
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malfet has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

In Python-3.9 and above `inspect.getsource` of a local class does not work if it was marked as default, see https://bugs.python.org/issue42666 #49617
Workaround by defining `make_global` function that programmatically acomplishes the same
@malfet malfet force-pushed the malfet/make-test_jit_py3-py3.9-compatible branch from 79391d9 to ff8869c Compare January 26, 2021 16:12
Copy link
Copy Markdown
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malfet has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Comment thread torch/testing/_internal/jit_utils.py
@t-vi
Copy link
Copy Markdown
Collaborator

t-vi commented Jan 26, 2021

Looks good to me, thank you!

At the risk of pointing out the obvious: An alternative to changing the how to make global could be to re-implement the old inspect way of finding the source code. This could be advantageous for users that run into this, but I wouldn't know if it is common. Finding class source code is superhacky in Python...

@facebook-github-bot
Copy link
Copy Markdown
Contributor

@malfet merged this pull request in a949d7b.

@malfet malfet deleted the malfet/make-test_jit_py3-py3.9-compatible branch January 26, 2021 20:53
@malfet
Copy link
Copy Markdown
Contributor Author

malfet commented Jan 26, 2021

An alternative to changing the how to make global could be to re-implement the old inspect way of finding the source code.

100% agree with you here, sounds like a good enhancement on the torch.jit frontend side.
@suo do you know if JIT needs source code for anything, but generating verbose error messages? (since JIT graph is generated from an AST, isn't it?)

@suo
Copy link
Copy Markdown
Member

suo commented Jan 27, 2021

The ast module requires source iirc. So I don't think there's a way to avoid retrieving source.

@t-vi: can you point me to the "old way" of doing inspect?

@SplitInfinity
Copy link
Copy Markdown

The ast module requires source iirc. So I don't think there's a way to avoid retrieving source.

Yes - we call ast.parse(source) to get the Python AST.

@t-vi
Copy link
Copy Markdown
Collaborator

t-vi commented Jan 27, 2021

The change that breaks our tests appears to be python/cpython#10307

Now, I will grant them that the old way of "just match a regexp" is far from ideal, either. I didn't try but now that I look more closely, maybe the correct solution would be to augment the _ClassFinder of the recent versions with scope awareness including global statments.
Given Python's release cycles, we might want some workaround for 3.9 and 3.10 at least, but in principle going through the AST seems much more robust than the old indent+regexp. Maybe we should just do our own AST visiting to get the right source lines.

When I tried to propose Python adding source file information to classes (for the benefit of Jupyter), some people on the mailing list suggested exposing the code object defining classes or so, which would give filename + line numbers. Having the file + line numbers would, of course, make this robust and trivial. The technical aspects of getting Python to have proper source annotations for their classes aren't hard, but the process of getting that type of thing into Python heavy-handed enough to make me give up: you'd need a PEP with performance considerations, breakage opportunities (someone's code will assume that anything with a code object is a function etc.).

soulitzer pushed a commit that referenced this pull request Jan 28, 2021
Summary:
In Python-3.9 and above `inspect.getsource` of a local class does not work if it was marked as default, see https://bugs.python.org/issue42666 #49617
Workaround by defining `make_global` function that programmatically accomplishes the same

Partially addresses issue raised in #49617

Pull Request resolved: #51088

Reviewed By: gmagogsfm

Differential Revision: D26069189

Pulled By: malfet

fbshipit-source-id: 7cf14b88ae5d2b95d2b0fd852717a9202b86356e
laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
Summary:
In Python-3.9 and above `inspect.getsource` of a local class does not work if it was marked as default, see https://bugs.python.org/issue42666 pytorch#49617
Workaround by defining `make_global` function that programmatically accomplishes the same

Partially addresses issue raised in pytorch#49617

Pull Request resolved: pytorch#51088

Reviewed By: gmagogsfm

Differential Revision: D26069189

Pulled By: malfet

fbshipit-source-id: 7cf14b88ae5d2b95d2b0fd852717a9202b86356e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed Merged oncall: jit Add this issue/PR to JIT oncall triage queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants