[ROCm][CI] last known good HIP patch by jeffdaily · Pull Request #158562 · pytorch/pytorch

jeffdaily · 2025-07-17T14:57:38Z

Fixes #ISSUE_NUMBER

cc @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2025-07-17T14:57:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158562

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 42 Unrelated Failures

As of commit 4e4a893 with merge base da4c7b4 ():

NEW FAILURES - The following jobs have failed:

inductor / unit-test / cuda12.8-py3.13-gcc9-sm86 / build (gh)
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 1, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh)
test_cuda_expandable_segments.py::TestCuda::test_invalid_status_for_legacy_api
pull / linux-jammy-py3.9-clang12 / test (dynamo_wrapped, 1, 3, linux.2xlarge) (gh)
'test/test_reductions.py::TestReductionsCPU::test_sum_all_cpu_float64'
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / build (gh)
ModuleNotFoundError: No module named 'setuptools.command.bdist_wheel'

CANCELLED JOB - The following job was cancelled. Please retry:

inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_timm_perf_rocm, 5, 5, linux.rocm.gpu.mi300.2) (gh)
##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 1, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 2, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 3, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 4, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 5, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 6, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 7, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-perf-nightly-rocm / rocm-py3_10-inductor-benchmark-test / test (inductor_torchbench_perf_rocm, 8, 8, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-periodic / cuda12.8-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
functorch_dp_cifar10
inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
mobilenetv3_large_100
inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
torchrec_dlrm
inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
mobilenetv3_large_100
inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
torchrec_dlrm
inductor-periodic / cuda12.8-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
torchrec_dlrm
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 1, 2, linux.8xlarge.amx) (gh) (similar failure)
detectron2_fcos_r_50_fpn
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.8xlarge.amx) (gh) (similar failure)
vision_maskrcnn
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_aot_inductor_freezing_torchbench, 1, 2, linux.8xlarge.amx) (gh) (similar failure)
basic_gnn_sage
inductor-periodic / rocm-py3_10-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-periodic / rocm-py3_10-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-periodic / rocm-py3_10-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-periodic / rocm-py3_10-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-periodic / rocm-py3_10-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
inductor-periodic / rocm-py3_10-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
/var/lib/jenkins/pytorch/fbgemm/src/EmbeddingSpMDM.cc:351:19: error: expected primary-expression before ‘const’
periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 1, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
periodic / linux-jammy-cuda12.4-py3.10-gcc11-sm89 / test (default, 1, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (similar failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
periodic / linux-jammy-cuda12.4-py3.10-gcc11-sm89 / test (default, 2, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (similar failure)
inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_frozen_fn
periodic / linux-jammy-cuda12.8-py3.10-gcc11 / test (nogpu_AVX512, 1, 3, lf.linux.4xlarge) (gh) (similar failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
periodic / linux-jammy-cuda12.8-py3.10-gcc11 / test (nogpu_NO_AVX2, 1, 2, lf.linux.4xlarge) (gh) (similar failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 1, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
periodic / linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) (gh) (similar failure)
distributed/checkpoint/test_hf_safetensor_e2e.py::TestDistributedHFSafetensorsConsolidation::test_consolidate_to_one_file
periodic-rocm-mi300 / linux-jammy-rocm-py3.10-mi300 / test (distributed, 2, 3, linux.rocm.gpu.mi300.4, module:rocm, oncall:distributed) (gh) (similar failure)
distributed/checkpoint/test_hf_safetensor_e2e.py::TestDistributedHFSafetensorsConsolidation::test_consolidate_to_one_file

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
torchrec_dlrm
pull / linux-jammy-cpu-py3.10-gcc11-bazel-test / build-and-test (default, 1, 1, linux.4xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-cuda12.8-py3.10-gcc11-sm89 / test (default, 1, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-py3.10-clang18-asan / test (default, 1, 6, linux.4xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-py3.13-clang12 / test (crossref, 1, 2, linux.2xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-py3.13-clang12 / test (default, 1, 5, linux.4xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-py3.9-clang12 / test (crossref, 1, 2, linux.2xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-py3.9-clang12 / test (default, 1, 5, linux.4xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, linux.2xlarge) (gh) (trunk failure)
[ FAILED ] TensorTest.TorchTensorCastComplexToRealErrorChecks

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh) (#153987)
MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

.ci/docker/common/install_rocm.sh

jithunnair-amd · 2025-07-18T04:03:21Z

Closing in favor of #158596

[DO NOT MERGE] last known good HIP patch

6153d60

pytorch-bot bot added the topic: not user facing topic category label Jul 17, 2025

facebook-github-bot added the module: rocm AMD GPU support for Pytorch label Jul 17, 2025

pytorchbot added the open source label Jul 17, 2025

jithunnair-amd added the ciflow/rocm Trigger "default" config CI on ROCm label Jul 17, 2025

use hash for clr checkout

8516d10

jeffdaily changed the title ~~[DO NOT MERGE] last known good HIP patch~~ [ROCm][CI] last known good HIP patch Jul 17, 2025

jeffdaily marked this pull request as ready for review July 17, 2025 19:48

fix cp lib

3d84cde

jithunnair-amd reviewed Jul 17, 2025

View reviewed changes

.ci/docker/common/install_rocm.sh Outdated Show resolved Hide resolved

jeffdaily added 2 commits July 17, 2025 21:22

swapped branches

f54b967

exclude 6.3

4e4a893

jithunnair-amd closed this Jul 18, 2025

jithunnair-amd deleted the rocm_hip_reset branch July 18, 2025 04:03

jithunnair-amd mentioned this pull request Jul 18, 2025

[ROCm][CI] Last known good HIP patch #158596

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ROCm][CI] last known good HIP patch#158562

[ROCm][CI] last known good HIP patch#158562
jeffdaily wants to merge 5 commits intopytorch:mainfrom
ROCm:rocm_hip_reset

jeffdaily commented Jul 17, 2025 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Jul 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

jithunnair-amd commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jeffdaily commented Jul 17, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158562

❌ 4 New Failures, 1 Cancelled Job, 42 Unrelated Failures

Uh oh!

Uh oh!

jithunnair-amd commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jeffdaily commented Jul 17, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 17, 2025 •

edited

Loading