[ROCm][CI] upgrade CI to ROCm 6.2 by jeffdaily · Pull Request #132555 · pytorch/pytorch

jeffdaily · 2024-08-02T22:28:01Z

cc @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @EikanWang @jgong5 @voznesenskym @penguinwu @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

pytorch-bot · 2024-08-02T22:28:04Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/132555

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 4 Unrelated Failures

As of commit c7ef4ff with merge base 1a86d8a ():

NEW FAILURES - The following jobs have failed:

Build libtorch docker images / build-docker-cuda (11.8) (gh)
periodic / ios-build-test / build (default, 1, 1, macos-14-xlarge, SIMULATOR, arm64, 1, 0, 1) (gh)
periodic / linux-focal-cuda11.8-py3.9-gcc9 / build (gh)
periodic / linux-focal-cuda11.8-py3.9-gcc9-experimental-split-build / build (gh)
periodic / parallelnative-linux-jammy-py3.9-gcc11 / test (default, 1, 3, lf.linux.2xlarge) (gh)
'Test'
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_compiled_optimizers.py::CompiledOptimizerParityTestsCUDA::test_correctness_ASGD_use_closure_True_cuda_float32
rocm / linux-focal-rocm6.2-py3.10 / test (default, 3, 6, linux.rocm.gpu.2) (gh)
inductor/test_pad_mm.py::PadMMTest::test_pad_mm_bf16

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-focal-rocm-n-1-py3) (gh) (detected as infra flaky with no log or failing log classifier)
docker-builds / docker-build (pytorch-linux-jammy-aarch64-py3.10-gcc11, linux.arm64.2xlarge) (gh) (detected as infra flaky with no log or failing log classifier)
inductor-periodic / cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_smoketest_perf, 1, 1, linux.gcp.a100) (gh) (detected as infra flaky with no runner)
pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh) (disabled by #134602)
test_transformers.py::TestSDPAPrivateUse1Only::test_scaled_dot_product_fused_attention_overrideable_backward

This comment was automatically generated by Dr. CI and updates every 15 minutes.

BLOrange-AMD · 2024-08-08T17:18:34Z

@pytorchbot label keep-going

BLOrange-AMD · 2024-08-08T17:20:42Z

@pytorchbot rebase

pytorchmergebot · 2024-08-08T17:22:19Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-08-08T17:22:22Z

Successfully rebased rocm62_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm62_ci && git pull --rebase)

malfet

LGTM

jeffdaily · 2024-09-20T17:36:52Z

@pytorchbot merge -f 'pulling trigger on rocm6.2 upgrade; unrelated test failures; rocm test failures flaky on 6.1 also'

pytorchmergebot · 2024-09-20T17:39:07Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Follow up to pytorch#132555.

jeffdaily · 2024-09-23T20:55:26Z

@pytorchbot cherry-pick --onto release/2.5 -c critical

pytorchbot · 2024-09-23T20:59:29Z

Cherry picking #132555

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 15dba021bb8fc2da1eae3a92cd159eeb8a346c04 returned non-zero exit code 1

Auto-merging .ci/docker/build.sh
Auto-merging .github/workflows/inductor-rocm.yml
CONFLICT (content): Merge conflict in .github/workflows/inductor-rocm.yml
Auto-merging .github/workflows/periodic.yml
CONFLICT (content): Merge conflict in .github/workflows/periodic.yml
Auto-merging .github/workflows/pull.yml
CONFLICT (content): Merge conflict in .github/workflows/pull.yml
Auto-merging .github/workflows/rocm.yml
CONFLICT (content): Merge conflict in .github/workflows/rocm.yml
Auto-merging .github/workflows/slow.yml
CONFLICT (content): Merge conflict in .github/workflows/slow.yml
Auto-merging .github/workflows/trunk.yml
CONFLICT (content): Merge conflict in .github/workflows/trunk.yml
Auto-merging test/inductor/test_torchinductor.py
Auto-merging torch/cuda/__init__.py
CONFLICT (content): Merge conflict in torch/cuda/__init__.py
Auto-merging torch/testing/_internal/common_methods_invocations.py
error: could not apply 15dba021bb... [ROCm][CI] upgrade CI to ROCm 6.2 (#132555)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Pull Request resolved: pytorch#132555 Approved by: https://github.com/pruthvistony, https://github.com/malfet

Follow up to #132555. Pull Request resolved: #136436 Approved by: https://github.com/jithunnair-amd, https://github.com/pruthvistony, https://github.com/atalman

Pull Request resolved: #132555 Approved by: https://github.com/pruthvistony, https://github.com/malfet

Follow up to pytorch#132555. Pull Request resolved: pytorch#136436 Approved by: https://github.com/jithunnair-amd, https://github.com/pruthvistony, https://github.com/atalman

jeffdaily added ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/rocm Trigger "default" config CI on ROCm ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Aug 2, 2024

jeffdaily requested a review from jithunnair-amd August 2, 2024 22:28

jeffdaily requested a review from a team as a code owner August 2, 2024 22:28

pytorch-bot bot added module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Aug 2, 2024

pytorchbot added the open source label Aug 2, 2024

pruthvistony approved these changes Aug 2, 2024

View reviewed changes

pytorch-bot bot added the NNC label Aug 5, 2024

jeffdaily mentioned this pull request Aug 5, 2024

Support for ROCm 6.2 #132570

Closed

pytorch-bot bot added the keep-going Don't stop on first failure, keep running tests until the end label Aug 8, 2024

jeffdaily added 2 commits August 8, 2024 17:22

[ROCm][CI] upgrade CI to ROCm 6.2

8d52393

lint

03f2e21

pytorchmergebot force-pushed the rocm62_ci branch from 0a53e63 to 03f2e21 Compare August 8, 2024 17:22

jeffdaily added 2 commits August 8, 2024 11:25

Merge branch 'main' into rocm62_ci

7fe8304

Merge branch 'main' into rocm62_ci

13a27b6

malfet approved these changes Aug 16, 2024

View reviewed changes

jeffdaily added 3 commits August 22, 2024 16:29

Merge branch 'main' into rocm62_ci

a0a5575

don't use rocm-llvm

7c0beaf

Merge branch 'main' into rocm62_ci

174ddd6

jithunnair-amd added ciflow/inductor ci-no-td Do not run TD on this PR labels Sep 6, 2024

jeffdaily added 2 commits September 12, 2024 16:27

build miopen from source

4c6ae40

partial revert

88b0045

jeffdaily added 3 commits September 12, 2024 17:42

Merge branch 'main' into rocm62_ci

6bdd885

work around miopen tolerances

0385f7c

work around amdsmi ROCm 6.2 deprecation in clock info

8533bb1

jeffdaily requested review from eqy and syed-ahmed as code owners September 13, 2024 16:19

pytorch-bot bot added the module: inductor label Sep 13, 2024

jeffdaily added 4 commits September 13, 2024 16:20

work around miopen tolerances

911a416

Merge branch 'main' into rocm62_ci

61513db

skip rocm 6.2 UT failure

2d1fd4e

Merge branch 'main' into rocm62_ci

c7ef4ff

jeffdaily requested a review from mruberry as a code owner September 18, 2024 21:07

pytorchmergebot added the merging label Sep 20, 2024

pytorchmergebot added the Merged label Sep 20, 2024

pytorchmergebot closed this in 15dba02 Sep 20, 2024

pytorchmergebot removed the merging label Sep 20, 2024

jeffdaily added a commit to ROCm/pytorch that referenced this pull request Sep 23, 2024

[ROCm] install_miopen.sh exit for ROCm >= 6.3

fb644de

Follow up to pytorch#132555.

jeffdaily mentioned this pull request Sep 23, 2024

[ROCm] install_miopen.sh exit for ROCm >= 6.3 #136436

Closed

jeffdaily added a commit to ROCm/pytorch that referenced this pull request Sep 23, 2024

[ROCm][CI] upgrade CI to ROCm 6.2 (pytorch#132555)

7ce41bb

Pull Request resolved: pytorch#132555 Approved by: https://github.com/pruthvistony, https://github.com/malfet

This was referenced Sep 23, 2024

[ROCm][CI] upgrade CI to ROCm 6.2 (#132555) #136467

Merged

[v.2.5.0] Release Tracker #135522

Closed

kit1980 pushed a commit that referenced this pull request Sep 25, 2024

[ROCm][CI] upgrade CI to ROCm 6.2 (#132555) (#136467)

7c550fe

Pull Request resolved: #132555 Approved by: https://github.com/pruthvistony, https://github.com/malfet

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ROCm][CI] upgrade CI to ROCm 6.2#132555

[ROCm][CI] upgrade CI to ROCm 6.2#132555
jeffdaily wants to merge 16 commits intopytorch:mainfrom
ROCm:rocm62_ci

jeffdaily commented Aug 2, 2024 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Aug 2, 2024 •

edited

Loading

Uh oh!

BLOrange-AMD commented Aug 8, 2024

Uh oh!

BLOrange-AMD commented Aug 8, 2024

Uh oh!

pytorchmergebot commented Aug 8, 2024

Uh oh!

pytorchmergebot commented Aug 8, 2024

Uh oh!

malfet left a comment

Uh oh!

jeffdaily commented Sep 20, 2024

Uh oh!

pytorchmergebot commented Sep 20, 2024

Uh oh!

jeffdaily commented Sep 23, 2024

Uh oh!

pytorchbot commented Sep 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

jeffdaily commented Aug 2, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/132555

❌ 7 New Failures, 4 Unrelated Failures

Uh oh!

BLOrange-AMD commented Aug 8, 2024

Uh oh!

BLOrange-AMD commented Aug 8, 2024

Uh oh!

pytorchmergebot commented Aug 8, 2024

Uh oh!

pytorchmergebot commented Aug 8, 2024

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

jeffdaily commented Sep 20, 2024

Uh oh!

pytorchmergebot commented Sep 20, 2024

Merge started

Uh oh!

jeffdaily commented Sep 23, 2024

Uh oh!

pytorchbot commented Sep 23, 2024

Cherry picking #132555

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jeffdaily commented Aug 2, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Aug 2, 2024 •

edited

Loading