Skip to content

[ROCm] Fix unit tests on CI#11191

Closed
iotamudelta wants to merge 534 commits intopytorch:masterfrom
ROCm:master
Closed

[ROCm] Fix unit tests on CI#11191
iotamudelta wants to merge 534 commits intopytorch:masterfrom
ROCm:master

Conversation

@iotamudelta
Copy link
Contributor

@iotamudelta iotamudelta commented Sep 3, 2018

Disables two of the unit tests in test_cuda that got introduced after test_cuda was enabled that fail on ROCm.

iotamudelta and others added 30 commits August 6, 2018 16:14
@iotamudelta
Merge branch 'master' into rocRAND_PR
b56ed34
@iotamudelta
Merge pull request #101 from jithunnair-amd/enable_optim_unit_tests
ae0336d 
Enable test_optim unit tests …
@iotamudelta
Merge remote-tracking branch 'upstream/master'
578f601
@iotamudelta
Merge remote-tracking branch 'upstream/master' into rocRAND_PR
7376e8e
@jithunnair-amd
Refactor unit test skip statements to use @skipIfRocm annotation
4ddc9ca
@iotamudelta
Merge pull request #102 from iotamudelta/master
1e6bf28 
Merge from upstream
@jithunnair-amd
Import skipIfRocm instead of TEST_WITH_ROCM from common
d963986
@iotamudelta
Merge remote-tracking branch 'rocm_upstream/master' into rocRAND_PR
b51b51e
@iotamudelta
Merge remote-tracking branch 'rocm_upstream/enableunittests' into roc…
e9c047e 
…RAND_PR

While there, add the remaining changes requested in upstream PR pytorch#10266
@iotamudelta
Do not set SHARED flag unconditionally here.
f1c36e6 
Reported by: bddqqp
@iotamudelta
Merge remote-tracking branch 'upstream/master'
8558486
@iotamudelta
Merge pull request #104 from iotamudelta/master
dee2f5b 
Merge from upstream
@iotamudelta
Merge pull request #103 from jithunnair-amd/skipIfRocm_refactor
1f3544f 
Refactor unit test skip statements to use @skipIfRocm annotation
@iotamudelta
Merge remote-tracking branch 'upstream/master'
4903af7
@iotamudelta
Merge branch 'master' into rocRAND_PR
d306547
@iotamudelta
Merge remote-tracking branch 'upstream/master'
410d539
@iotamudelta
Merge branch 'master' into rocRAND_PR
4692427
@iotamudelta
Merge pull request #105 from iotamudelta/master
fa99099 
Merge from upstream
@iotamudelta
Merge remote-tracking branch 'upstream/master'
eb5975a
@iotamudelta
Merge branch 'master' into rocRAND_PR
ca94a4a
@mwootton
Update native_functions for MIOpen
78eea43
@mwootton
Merge branch 'master' into miopen_integration
3774e54
@mwootton
Fix build on cuda/cudnn platforms
ed19fc6
@iotamudelta
Merge pull request #41 from iotamudelta/rocRAND_PR
f025a43 
Replace hcRNG with rocRAND.
@iotamudelta
Merge remote-tracking branch 'upstream/master'
cde2069
@iotamudelta
First patch to remove hipblas and replace with direct rocblas calls.
f363f87
@iotamudelta
Fix typo.
4819f55
@iotamudelta
rocBLAS supports strided batched gemms.
67f77f0
@iotamudelta
For ROCm, wire batched through to strided batched.
f6d3225
@iotamudelta
Use __HIP_PLATFORM_HCC__ instead of USE_ROCM macro.
00ec9d5
jithunnair-amd and others added 15 commits August 29, 2018 10:55
@jithunnair-amd
Merge branch 'skip_tests' of https://github.com/jithunnair-amd/pytorch …
314f37d 
…into skip_tests
@iotamudelta
Merge branch 'master' into ifu
0fbc1c0
@iotamudelta
Merge remote-tracking branch 'upstream/master' into ifu
67097e9
@iotamudelta
Fix typo.
fbfe994
@lcskrishna
fixed merge conflicts
58426b3
@jithunnair-amd
Merge pull request #1 from lcskrishna/cl/merge-conflicts
be8e093 
fixed merge conflicts.
@iotamudelta
Merge pull request #163 from iotamudelta/rocm19
d54243e 
Fix typo.
@jithunnair-amd
Skip test_jit.py TestEndToEndHybridFrontendModels.test_super_resoluti…
f53f3b9 
…on due to recently observed hang
@iotamudelta
Merge pull request #161 from jithunnair-amd/skip_tests
1245c83 
Skip KLDivLoss_cuda tests due to hang
@iotamudelta
Merge pull request #157 from iotamudelta/ifu
b40c0bb 
Merge from upstream
@iotamudelta
Merge remote-tracking branch 'upstream/master' into ifu
c69f5da
@iotamudelta
Merge pull request #164 from iotamudelta/ifu
9d9118a 
Merge from upstream
@iotamudelta
Merge remote-tracking branch 'upstream/master' into ifu
f543557
@iotamudelta
Fails on ROCm currently.
c9cf2c9
@iotamudelta
Merge pull request #167 from iotamudelta/ifu
59d1430 
Merge from upstream
facebook-github-bot
facebook-github-bot reviewed Sep 3, 2018
View reviewed changes
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ssnl
Copy link
Collaborator

ssnl commented Sep 3, 2018

I'm really confused why these tests fail with

02:00:13 ======================================================================
02:00:13 ERROR: test_max_with_inf (__main__.TestCuda)
02:00:13 ----------------------------------------------------------------------
02:00:13 Traceback (most recent call last):
02:00:13   File "/var/lib/jenkins/workspace/test/common.py", line 251, in wrapper
02:00:13     method(*args, **kwargs)
02:00:13   File "test_cuda.py", line 1700, in test_max_with_inf
02:00:13     TestTorch._test_max_with_inf(self, (torch.half, torch.float, torch.double), 'cuda')
02:00:13   File "/var/lib/jenkins/workspace/test/test_torch.py", line 800, in _test_max_with_inf
02:00:13     self.assertTrue(torch.all(torch.max(a, dim=1)[0] == inf).item())
02:00:13 RuntimeError: Expected object of scalar type Half but got scalar type Long for argument #0 'max'

Any ideas?

@iotamudelta
Copy link
Contributor Author

iotamudelta commented Sep 3, 2018

@ssnl unclear. We are working on unit test pass rate ATM. There are a few things that could interfere here: a) we are aware of a compiler bug (fixed in #11198 ) that can cause hangs and crashes, b) we are also aware of a few tests (min/max in particular) that succeed on our nodes but fail on the CI (we are looking into this).

PenghuiCheng pushed a commit to PenghuiCheng/pytorch that referenced this pull request Sep 11, 2018
@iotamudelta @PenghuiCheng
Fix unit tests on CI (pytorch#11191)
76c4d6d 
Summary:
Disables two of the  unit tests in test_cuda that got introduced after test_cuda was enabled that fail on ROCm.
Pull Request resolved: pytorch#11191

Differential Revision: D9628702

Pulled By: ezyang

fbshipit-source-id: 4c298c728f42bb43d39b57967aa3e44385980265
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

@ezyang ezyang ezyang approved these changes

@apaszke apaszke Awaiting requested review from apaszke

@colesbury colesbury Awaiting requested review from colesbury

@gchanan gchanan Awaiting requested review from gchanan

@soumith soumith Awaiting requested review from soumith

@zdevito zdevito Awaiting requested review from zdevito

Assignees

No one assigned

Labels

open source

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@iotamudelta @ssnl @ezyang @facebook-github-bot @jithunnair-amd @mwootton @lcskrishna