Skip to content

Fix broken FX tests#83187

Closed
datumbox wants to merge 1 commit intomasterfrom
datumbox-patch-1
Closed

Fix broken FX tests#83187
datumbox wants to merge 1 commit intomasterfrom
datumbox-patch-1

Conversation

@datumbox
Copy link
Copy Markdown
Contributor

@datumbox datumbox commented Aug 10, 2022

Resolves breakages observed at #82560

Context:
The current FX tests assume that every public method under torchvision.models is a model builder method. To get a list of those methods, they query the __dict__ attribute of the module. Unfortunately this assumption is not true and the tests already contain some workarounds to filter some methods. A better approach would be to query TorchVision for all of its available models under a specific module. This is exactly what the new Registration API can help us do and that's what we use in this PR.

@datumbox datumbox requested a review from atalman August 10, 2022 17:32
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 10, 2022 •
edited
Loading

🔗 Helpful links

✅ No Failures (0 Pending)

As of commit 46bbbf1 (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 10, 2022

@datumbox has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@datumbox
Copy link
Copy Markdown
Contributor Author

datumbox commented Aug 10, 2022

So the tests fail with:

Traceback (most recent call last):
  File "test_fx.py", line 4206, in <module>
    TestVisionTracing.generate_tests()
  File "test_fx.py", line 4200, in generate_tests
    cls.generate_classification_tests()
  File "test_fx.py", line 4164, in generate_classification_tests
    for k in torchvision_models.list_models(module=torchvision_models):
AttributeError: module 'torchvision.models' has no attribute 'list_models'
Traceback (most recent call last):
  File "test/run_test.py", line 974, in <module>
    main()
  File "test/run_test.py", line 952, in main
    raise RuntimeError(err_message)
RuntimeError: test_fx failed!

But that's because we install an old version of TorchVision that doesn't contain the necessary list_models() method:

Successfully installed torchvision-0.14.0a0+1a1d509

The specific hashcode is the one last updated as seen at #82560. So basically we are in the situation where the test now fails because we can't install the TorchVision version that actually contains the new functionality for solving the issue. How do you fix this normally?

@datumbox datumbox requested a review from a team as a code owner August 10, 2022 20:36
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 10, 2022

This pull request was exported from Phabricator. Differential Revision: D38583882

@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented Aug 10, 2022

Update the torchvision hash as part of this PR. .github/ci_commit_pins/vision.txt

@datumbox
Copy link
Copy Markdown
Contributor Author

datumbox commented Aug 10, 2022

@ezyang Yeap that's what I did. Takes ages to export diffs today. Thanks for the tip.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 10, 2022

This pull request was exported from Phabricator. Differential Revision: D38583882

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 10, 2022

This pull request was exported from Phabricator. Differential Revision: D38583882

@datumbox @facebook-github-bot
Fix broken FX tests (#83187)
46bbbf1 
Summary:
Resolves [breakages](https://github.com/pytorch/pytorch/runs/7762125339?check_suite_focus=true) observed at #82560

Context:
The current FX tests assume that every public method under `torchvision.models` is a model builder method. To get a list of those methods, they query the `__dict__` attribute of the module. Unfortunately this assumption is not true and the tests already contain some workarounds to filter some methods. A better approach would be to query TorchVision for all of its available models under a specific module. This is exactly what the new Registration API can help us do and that's what we use in this PR.

Pull Request resolved: #83187

Test Plan: buck1 test mode/dev //caffe2/test:fx

Reviewed By: ezyang

Differential Revision: D38583882

Pulled By: datumbox

fbshipit-source-id: a2b98596928f95fba7e4f34d047e0602f0a6c1f9
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 10, 2022

This pull request was exported from Phabricator. Differential Revision: D38583882

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Aug 11, 2022

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Aug 11, 2022

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 11, 2022

Hey @datumbox.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit that referenced this pull request Aug 11, 2022
@datumbox @facebook-github-bot
Fix broken FX tests (#83187)
38c8b8e 
Summary:
Resolves [breakages](https://github.com/pytorch/pytorch/runs/7762125339?check_suite_focus=true) observed at #82560

Context:
The current FX tests assume that every public method under `torchvision.models` is a model builder method. To get a list of those methods, they query the `__dict__` attribute of the module. Unfortunately this assumption is not true and the tests already contain some workarounds to filter some methods. A better approach would be to query TorchVision for all of its available models under a specific module. This is exactly what the new Registration API can help us do and that's what we use in this PR.

Pull Request resolved: #83187

Test Plan: buck1 test mode/dev //caffe2/test:fx

Reviewed By: ezyang

Differential Revision: D38583882

Pulled By: datumbox

fbshipit-source-id: 65ac636afc9898ca6edaddfd085db69ec40fd34d
@datumbox datumbox deleted the datumbox-patch-1 branch August 11, 2022 07:47
@datumbox datumbox added module: tests Issues related to tests (not the torch.testing module) fx labels Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ezyang ezyang ezyang approved these changes

@atalman atalman Awaiting requested review from atalman

@SherlockNoMad SherlockNoMad Awaiting requested review from SherlockNoMad

Assignees

No one assigned

Labels

cla signed fx Merged module: tests Issues related to tests (not the torch.testing module)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@datumbox @facebook-github-bot @ezyang @pytorchmergebot