Allow ROCm CI to use non-default stream. by jeffdaily · Pull Request #48424 · pytorch/pytorch

jeffdaily · 2020-11-24T20:18:27Z

Revert #26394. Fixes #27356. Not all MIOpen handles were setting their stream to the current stream prior to running the op.

jeffdaily · 2020-11-24T20:19:21Z

This PR will test if #27356 is still really an issue. It's been a year since that issue was raised.

dr-ci · 2020-11-24T20:30:39Z

💊 CI failures summary and remediations

As of commit 5b01d9f (more details on the Dr. CI page):

3/3 failures introduced in this PR

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_xla_linux_bionic_py3_6_clang9_build (1/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/verbatim-sources/job-specs/pytorch-job-specs.yml 
Auto-merging .circleci/verbatim-sources/job-specs/pytorch-job-specs.yml 
CONFLICT (add/add): Merge conflict in .circleci/docker/build.sh 
Auto-merging .circleci/docker/build.sh 
CONFLICT (add/add): Merge conflict in .circleci/config.yml 
Auto-merging .circleci/config.yml 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/docker_definitions.py 
Auto-merging .circleci/cimodel/data/simple/docker_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
Automatic merge failed; fix conflicts and then commit the result.

pytorch_linux_xenial_py3_6_gcc5_4_build (2/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/verbatim-sources/job-specs/pytorch-job-specs.yml 
Auto-merging .circleci/verbatim-sources/job-specs/pytorch-job-specs.yml 
CONFLICT (add/add): Merge conflict in .circleci/docker/build.sh 
Auto-merging .circleci/docker/build.sh 
CONFLICT (add/add): Merge conflict in .circleci/config.yml 
Auto-merging .circleci/config.yml 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/docker_definitions.py 
Auto-merging .circleci/cimodel/data/simple/docker_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
Automatic merge failed; fix conflicts and then commit the result.

pytorch_linux_xenial_cuda9_2_cudnn7_py3_gcc5_4_build (3/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/verbatim-sources/job-specs/pytorch-job-specs.yml 
Auto-merging .circleci/verbatim-sources/job-specs/pytorch-job-specs.yml 
CONFLICT (add/add): Merge conflict in .circleci/docker/build.sh 
Auto-merging .circleci/docker/build.sh 
CONFLICT (add/add): Merge conflict in .circleci/config.yml 
Auto-merging .circleci/config.yml 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/docker_definitions.py 
Auto-merging .circleci/cimodel/data/simple/docker_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
Automatic merge failed; fix conflicts and then commit the result.

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 20 times.

jeffdaily · 2020-11-24T23:41:03Z

@pytorchbot retest this please

codecov · 2020-11-24T23:41:41Z

Codecov Report

Merging #48424 (2e286e2) into master (3b25af0) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #48424      +/-   ##
==========================================
- Coverage   80.83%   80.83%   -0.01%     
==========================================
  Files        1860     1860              
  Lines      200668   200668              
==========================================
- Hits       162220   162219       -1     
- Misses      38448    38449       +1

jeffdaily · 2020-11-25T00:00:30Z

@pytorchbot retest this please

jeffdaily · 2020-11-30T21:20:12Z

@pytorchbot retest this please

Revert pytorch#26394. Fix pytorch#27356.

This removes the util function setMIOpenStreamToCurrent() in favor of always setting the current stream to the MIOpen handle when requested.

jeffdaily · 2020-12-04T22:10:02Z

@jithunnair-amd could you also review? Thanks.

jeffdaily · 2020-12-07T16:31:29Z

CI failures due to upstream, not this PR. Retesting.

jeffdaily · 2020-12-07T16:31:38Z

@pytorchbot retest this please

jeffdaily · 2020-12-07T20:56:11Z

@mruberry I'm happy to finally resolve this year-old issue. Thanks in advance for your review.

sunway513

LGTM, thanks @jeffdaily.

mruberry · 2020-12-08T09:23:56Z

Hey @jeffdaily, this looks OK to me. Do you want to wait for @jithunnair-amd?

jithunnair-amd · 2020-12-08T13:43:17Z

LGTM too

mruberry

Cool!

facebook-github-bot

@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-12-10T19:16:22Z

@mruberry merged this pull request in d5c4a80.

facebook-github-bot · 2020-12-10T19:16:39Z

@mruberry merged this pull request in d5c4a80.

This reverts commit d5c4a80.

Summary: Revert pytorch#26394. Fixes pytorch#27356. Not all MIOpen handles were setting their stream to the current stream prior to running the op. Pull Request resolved: pytorch#48424 Reviewed By: H-Huang Differential Revision: D25420384 Pulled By: mruberry fbshipit-source-id: 051683ba9e3d264b71162bd344031a0c58bf6a41

jeffdaily added the module: rocm AMD GPU support for Pytorch label Nov 24, 2020

facebook-github-bot added the cla signed label Nov 24, 2020

pytorchbot added the open source label Nov 24, 2020

jeffdaily marked this pull request as ready for review November 24, 2020 23:59

jeffdaily marked this pull request as draft November 25, 2020 00:06

jeffdaily force-pushed the rocm_tests_on_non_default_stream branch from e847edf to 503de05 Compare December 1, 2020 18:45

Allow ROCm CI to use non-default stream.

2e286e2

Revert pytorch#26394. Fix pytorch#27356.

jeffdaily force-pushed the rocm_tests_on_non_default_stream branch from 503de05 to 2e286e2 Compare December 2, 2020 17:07

always set current stream when getting an MIOpen handle

0d6afd5

This removes the util function setMIOpenStreamToCurrent() in favor of always setting the current stream to the MIOpen handle when requested.

jeffdaily requested a review from sunway513 December 4, 2020 22:08

jeffdaily marked this pull request as ready for review December 4, 2020 23:07

jeffdaily marked this pull request as draft December 4, 2020 23:42

jeffdaily added 2 commits December 5, 2020 00:47

update MIOpen handle logic based on CuDNN handle pool

27619a3

enable test_cudnn_multiple_threads_same_device

5b01d9f

jeffdaily marked this pull request as ready for review December 7, 2020 20:44

jeffdaily requested a review from mruberry December 7, 2020 20:47

sunway513 approved these changes Dec 7, 2020

View reviewed changes

jithunnair-amd approved these changes Dec 8, 2020

View reviewed changes

zhangguanheng66 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Dec 8, 2020

mruberry approved these changes Dec 9, 2020

View reviewed changes

facebook-github-bot reviewed Dec 9, 2020

View reviewed changes

facebook-github-bot closed this in d5c4a80 Dec 10, 2020

facebook-github-bot added the Merged label Dec 10, 2020

jeffdaily added a commit to ROCm/pytorch that referenced this pull request Feb 17, 2021

Revert "Allow ROCm CI to use non-default stream. (pytorch#48424)"

ba490b6

This reverts commit d5c4a80.

Conversation

jeffdaily commented Nov 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffdaily commented Nov 24, 2020

Uh oh!

dr-ci Bot commented Nov 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 3 new failures recognized by patterns

pytorch_xla_linux_bionic_py3_6_clang9_build (1/3)

pytorch_linux_xenial_py3_6_gcc5_4_build (2/3)

pytorch_linux_xenial_cuda9_2_cudnn7_py3_gcc5_4_build (3/3)

Uh oh!

jeffdaily commented Nov 24, 2020

Uh oh!

codecov Bot commented Nov 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jeffdaily commented Nov 25, 2020

Uh oh!

jeffdaily commented Nov 30, 2020

Uh oh!

jeffdaily commented Dec 4, 2020

Uh oh!

jeffdaily commented Dec 7, 2020

Uh oh!

jeffdaily commented Dec 7, 2020

Uh oh!

jeffdaily commented Dec 7, 2020

Uh oh!

sunway513 left a comment

Choose a reason for hiding this comment

Uh oh!

mruberry commented Dec 8, 2020

Uh oh!

jithunnair-amd commented Dec 8, 2020

Uh oh!

mruberry left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Dec 10, 2020

Uh oh!

facebook-github-bot commented Dec 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jeffdaily commented Nov 24, 2020 •

edited

Loading

dr-ci Bot commented Nov 24, 2020 •

edited

Loading

codecov Bot commented Nov 24, 2020 •

edited

Loading