[ROCm][CI] Upgrade ROCm to 7.0 by pruthvistony · Pull Request #163140 · pytorch/pytorch

pruthvistony · 2025-09-17T03:23:56Z

Upgrade all the ROCm docker image to ROCm 7.0 release version.

cc @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2025-09-17T03:24:01Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163140

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 19 Unrelated Failures

As of commit da7607a with merge base 11b4c0e ():

NEW FAILURES - The following jobs have failed:

inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (aot_eager_timm, 1, 2, linux.rocm.gpu.gfx942.1) (gh)
dla102
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (dynamic_aot_eager_timm, 1, 2, linux.rocm.gpu.gfx942.1) (gh)
eca_botnext26ts_256
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (dynamo_eager_timm, 1, 2, linux.rocm.gpu.gfx942.1) (gh)
dla102
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 1, 10, linux.s390x) (gh)
'test/test_tensor_creation_ops.py::TestTensorCreationCPU::test_cat_large_tensor_cpu_float32'
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 6, 10, linux.s390x) (gh)
test_nestedtensor.py::TestNestedTensorSubclassCPU::test_sdpa_cpu_float16
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 9, 10, linux.s390x) (gh)
test_nestedtensor.py::TestNestedTensorSubclassCPU::test_sdpa_cpu_float32

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-periodic / inductor-smoke-test / test (inductor_torchbench_smoketest_perf, 1, 1, linux.aws.a100) (gh) (similar failure)
Process completed with exit code 1.
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (aot_eager_torchbench, 2, 2, linux.rocm.gpu.gfx942.1) (gh) (similar failure)
timm_regnet
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (dynamic_aot_eager_torchbench, 2, 2, linux.rocm.gpu.gfx942.1) (gh) (similar failure)
timm_regnet
inductor-periodic / rocm-periodic-dynamo-benchmarks-test / test (dynamo_eager_torchbench, 2, 2, linux.rocm.gpu.gfx942.1) (gh) (similar failure)
timm_regnet
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 2, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 3, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 4, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 6, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
inductor/test_multi_kernel.py::MultiKernelTest::test_triton_relu_fused_gemm
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 2, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_64_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 3, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_65_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 4, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_128_out_features_65_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 5, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_1024_out_features_65_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 6, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
inductor/test_cuda_select_algorithm.py::TestSelectAlgorithmCudaCUDA::test_int8_woq_mm_cuda_batch_size_32_mid_dim_8_in_features_144_out_features_1024_cuda_bfloat16
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 7, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (disabled by #156736)
dynamo/test_repros.py::ReproTests::test_bitwise_print_precedence

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 1, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#163657)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 2, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#163657)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 3, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#163657)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 4, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#163657)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 5, 5, lf.linux.4xlarge.nvidia.gpu, unstable) (gh) (#163657)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jeffdaily · 2025-09-22T20:37:10Z

@pytorchbot rebase

pytorchmergebot · 2025-09-22T20:38:42Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-09-22T20:38:45Z

Successfully rebased rocm70_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm70_ci && git pull --rebase)

jeffdaily · 2025-09-24T16:03:16Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-09-24T16:04:47Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-09-24T16:04:50Z

Successfully rebased rocm70_ci onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout rocm70_ci && git pull --rebase)

jeffdaily · 2025-09-26T05:52:25Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-09-26T05:54:09Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-09-26T05:54:12Z

Successfully rebased rocm70_ci onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout rocm70_ci && git pull --rebase)

jeffdaily · 2025-09-30T02:21:06Z

@pytorchbot merge -f "with the exception of flaky dynamo benchmarks, all ROCm CI is clean; only ROCm CI changes in this PR"

pytorchmergebot · 2025-09-30T02:22:38Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pruthvistony requested review from a team and jeffdaily as code owners September 17, 2025 03:23

pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Sep 17, 2025

This comment was marked as outdated.

Sign in to view

pytorchbot added the open source label Sep 17, 2025

jeffdaily changed the title ~~[ROCm][CI] Upgrade ROCm to 7.0~~ [ROCm][CI] Upgrade ROCm to 7.0.1 Sep 18, 2025

zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Sep 18, 2025

jeffdaily approved these changes Sep 18, 2025

View reviewed changes

jeffdaily added the ciflow/inductor-periodic label Sep 18, 2025

pytorchmergebot force-pushed the rocm70_ci branch from e769a9f to 050ecd5 Compare September 22, 2025 20:38

jithunnair-amd mentioned this pull request Sep 23, 2025

DISABLED test_select_SparseBSC_int32_cuda_bfloat16 (__main__.TestSparseCSRCUDA) #160784

Closed

pytorchmergebot force-pushed the rocm70_ci branch from 050ecd5 to 92df495 Compare September 24, 2025 16:04

jeffdaily added 3 commits September 26, 2025 05:54

[ROCm][CI] Upgrade ROCm to 7.0

5cce3f6

upgrade to rocm 7.0.1, remove prior alpha case

d071aa3

update ucx and ucc to tip commits to pick up rocm changes

31a1418

pytorchmergebot force-pushed the rocm70_ci branch from 92df495 to 31a1418 Compare September 26, 2025 05:54

rocm 7.0.1 no userspace changes

8568041

jeffdaily changed the title ~~[ROCm][CI] Upgrade ROCm to 7.0.1~~ [ROCm][CI] Upgrade ROCm to 7.0 Sep 26, 2025

jeffdaily added 2 commits September 26, 2025 16:54

update MAGMA_VERSION for docker image builds

48a9f3a

skip regressed tests

da7607a

pytorchmergebot added the merging label Sep 30, 2025

pytorchmergebot added the Merged label Sep 30, 2025

pytorchmergebot closed this in b7419b9 Sep 30, 2025

pytorchmergebot removed the merging label Sep 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ROCm][CI] Upgrade ROCm to 7.0#163140

[ROCm][CI] Upgrade ROCm to 7.0#163140
pruthvistony wants to merge 6 commits intopytorch:mainfrom
ROCm:rocm70_ci

pruthvistony commented Sep 17, 2025 •

edited by jeffdaily

Loading

Uh oh!

pytorch-bot bot commented Sep 17, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

jeffdaily commented Sep 22, 2025

Uh oh!

pytorchmergebot commented Sep 22, 2025

Uh oh!

pytorchmergebot commented Sep 22, 2025

Uh oh!

jeffdaily commented Sep 24, 2025

Uh oh!

pytorchmergebot commented Sep 24, 2025

Uh oh!

pytorchmergebot commented Sep 24, 2025

Uh oh!

jeffdaily commented Sep 26, 2025

Uh oh!

pytorchmergebot commented Sep 26, 2025

Uh oh!

pytorchmergebot commented Sep 26, 2025

Uh oh!

jeffdaily commented Sep 30, 2025

Uh oh!

pytorchmergebot commented Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

pruthvistony commented Sep 17, 2025 • edited by jeffdaily Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/163140

❌ 6 New Failures, 19 Unrelated Failures

Uh oh!

This comment was marked as outdated.

jeffdaily commented Sep 22, 2025

Uh oh!

pytorchmergebot commented Sep 22, 2025

Uh oh!

pytorchmergebot commented Sep 22, 2025

Uh oh!

jeffdaily commented Sep 24, 2025

Uh oh!

pytorchmergebot commented Sep 24, 2025

Uh oh!

pytorchmergebot commented Sep 24, 2025

Uh oh!

jeffdaily commented Sep 26, 2025

Uh oh!

pytorchmergebot commented Sep 26, 2025

Uh oh!

pytorchmergebot commented Sep 26, 2025

Uh oh!

jeffdaily commented Sep 30, 2025

Uh oh!

pytorchmergebot commented Sep 30, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pruthvistony commented Sep 17, 2025 •

edited by jeffdaily

Loading

pytorch-bot bot commented Sep 17, 2025 •

edited

Loading