[ONNX] Fix for constant folding flaky tests by neginraoof · Pull Request #32546 · pytorch/pytorch

neginraoof · 2020-01-23T19:47:31Z

Fix for constant folding flaky tests
Looks like the constant folding test modules are sometimes exported with ONNX_ATEN op export type, which is causing the CI failures.
I'm unable to repro this issue locally, but my guess is that the op export param is being overwritten on CI build at some point.
This PR sets the op export type and hopefully fixes the issue.

kostmo · 2020-01-23T19:54:59Z

💊 CircleCI build failures summary and remediations

As of commit 3627aad:

1/3 broken upstream at merge base 957a07f since Jan 27
Please rebase on the viable/strict branch (expand for instructions)

If your commit is newer than viable/strict, you can try basing on an older, stable commit:
```
git fetch origin viable/strict
git rebase --onto viable/strict $(git merge-base origin/master HEAD)
```
If your commit is older than viable/strict:
```
git fetch origin viable/strict
git rebase viable/strict
```
Check out the recency history of this "viable master" tracking branch.
1/3 failures introduced in this PR
1/3 recognized as flaky ❄️
- Re-run these jobs?

Detailed failure analysis

One may explore the probable reasons each build failed interactively on the Dr. CI website.

🕵️ 1 new failure recognized by patterns

The following build failures do not appear to be due to upstream breakage:

pytorch_linux_xenial_py3_6_gcc5_4_test (1/1)

Step: "Test" (full log | pattern match details)

Jan 27 17:06:27 RuntimeError: test_quantization failed!

Jan 27 17:06:27 Ran 36 tests in 50.351s 
Jan 27 17:06:27  
Jan 27 17:06:27 FAILED (errors=1, skipped=1) 
Jan 27 17:06:27  
Jan 27 17:06:27 Generating XML reports... 
Jan 27 17:06:27 Traceback (most recent call last): 
Jan 27 17:06:27   File "test/run_test.py", line 456, in <module> 
Jan 27 17:06:27     main() 
Jan 27 17:06:27   File "test/run_test.py", line 449, in main 
Jan 27 17:06:27     raise RuntimeError(message) 
Jan 27 17:06:27 RuntimeError: test_quantization failed! 
Jan 27 17:06:27 + cleanup 
Jan 27 17:06:27 + retcode=1 
Jan 27 17:06:27 + set +x 
Jan 27 17:06:27 =================== sccache compilation log =================== 
Jan 27 17:06:27 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Jan 27 17:06:27 Compile requests                  7 
Jan 27 17:06:27 Compile requests executed         6 
Jan 27 17:06:27 Cache hits                        0 
Jan 27 17:06:27 Cache misses                      6 
Jan 27 17:06:27 Cache timeouts                    0

❄️ 1 failure recognized as flaky

The following build failures have been detected as flaky and may not be your fault:

pytorch_linux_xenial_cuda10_1_cudnn7_py3_gcc7_test (1/1)

Step: "Test" (full log | pattern match details) ❄️

Jan 27 17:53:28 AssertionError: 10 not less than or equal to 1e-05 :

Jan 27 17:53:28 ---------------------------------------------------------------------- 
Jan 27 17:53:28 Traceback (most recent call last): 
Jan 27 17:53:28   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper 
Jan 27 17:53:28     self._join_processes(fn) 
Jan 27 17:53:28   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 255, in _join_processes 
Jan 27 17:53:28     self._check_return_codes(elapsed_time) 
Jan 27 17:53:28   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 275, in _check_return_codes 
Jan 27 17:53:28     self.assertEqual(p.exitcode, first_process.exitcode) 
Jan 27 17:53:28   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 892, in assertEqual 
Jan 27 17:53:28     super(TestCase, self).assertLessEqual(abs(x - y), prec, message) 
Jan 27 17:53:28 AssertionError: 10 not less than or equal to 1e-05 :  
Jan 27 17:53:28  
Jan 27 17:53:28 ---------------------------------------------------------------------- 
Jan 27 17:53:28 Ran 117 tests in 150.248s 
Jan 27 17:53:28  
Jan 27 17:53:28 FAILED (failures=1, skipped=7) 
Jan 27 17:53:28  
Jan 27 17:53:28 Generating XML reports... 
Jan 27 17:53:29 Traceback (most recent call last): 
Jan 27 17:53:29   File "test/run_test.py", line 456, in <module> 
Jan 27 17:53:29     main()

🚧 1 upstream failure recognized by patterns:

These builds matched patterns, but were probably caused by upstream breakages:

pytorch_macos_10_13_py3_test from Jan 27

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 6 times.

BowenBao

LGTM, thanks!

…oof/fixConstFoldingFlaky

facebook-github-bot

@houseroad has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

neginraoof · 2020-01-29T17:24:52Z

@houseroad Can we merge this? Thanks!

houseroad

Hopefully this will work.

facebook-github-bot · 2020-02-04T01:12:13Z

@houseroad merged this pull request in 5c019fe.

Summary: Fix for constant folding flaky tests Looks like the constant folding test modules are sometimes exported with ONNX_ATEN op export type, which is causing the CI failures. I'm unable to repro this issue locally, but my guess is that the op export param is being overwritten on CI build at some point. This PR sets the op export type and hopefully fixes the issue. Pull Request resolved: pytorch#32546 Reviewed By: hl475 Differential Revision: D19606919 Pulled By: houseroad fbshipit-source-id: 31793d6857bbbf99b43b4a7c22a045a56ae19e44

fix for flaky tests

388bd3e

pytorchbot added the open source label Jan 23, 2020

Trigger build

0e99d1d

BowenBao approved these changes Jan 23, 2020

View reviewed changes

Merge branch 'master' of https://github.com/pytorch/pytorch into nera…

3627aad

…oof/fixConstFoldingFlaky

facebook-github-bot reviewed Jan 28, 2020

View reviewed changes

houseroad approved these changes Feb 3, 2020

View reviewed changes

facebook-github-bot closed this in 5c019fe Feb 3, 2020

facebook-github-bot added the merged label Feb 4, 2020

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ONNX] Fix for constant folding flaky tests#32546

[ONNX] Fix for constant folding flaky tests#32546
neginraoof wants to merge 3 commits intopytorch:masterfrom
neginraoof:neraoof/fixConstFoldingFlaky

neginraoof commented Jan 23, 2020 •

edited

Loading

Uh oh!

kostmo commented Jan 23, 2020 •

edited

Loading

Uh oh!

BowenBao left a comment

Uh oh!

facebook-github-bot left a comment

Uh oh!

neginraoof commented Jan 29, 2020

Uh oh!

houseroad left a comment

Uh oh!

facebook-github-bot commented Feb 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

neginraoof commented Jan 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kostmo commented Jan 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CircleCI build failures summary and remediations

Detailed failure analysis

🕵️ 1 new failure recognized by patterns

pytorch_linux_xenial_py3_6_gcc5_4_test (1/1)

❄️ 1 failure recognized as flaky

pytorch_linux_xenial_cuda10_1_cudnn7_py3_gcc7_test (1/1)

🚧 1 upstream failure recognized by patterns:

Uh oh!

BowenBao left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

neginraoof commented Jan 29, 2020

Uh oh!

houseroad left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Feb 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

neginraoof commented Jan 23, 2020 •

edited

Loading

kostmo commented Jan 23, 2020 •

edited

Loading