Enable target-determination (TD) for ROCm CI by jithunnair-amd · Pull Request #156545 · pytorch/pytorch

jithunnair-amd · 2025-06-21T13:12:16Z

Target determination sorts the tests in a PR CI run based on heuristics about which tests are more relevant to the PR's changes. This can help provide faster CI signal as well as help alleviate capacity concerns as job durations should decrease due to catching failures earlier.

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2025-06-21T13:12:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156545

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures, 23 Unrelated Failures

As of commit e1ed5e2 with merge base 1cfdcb9 ():

NEW FAILURES - The following jobs have failed:

periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 2, 8, linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh)
'test/inductor/test_perf.py::InplacingTests::test_inplace_custom_op'
rocm / linux-jammy-rocm-py3.10 / test (default, 5, 6, linux.rocm.gpu.2) (gh)
'test/inductor/test_perf.py::InplacingTests::test_inplace_custom_op'
rocm-mi300 / linux-jammy-rocm-py3.10-mi300 / test (default, 3, 6, linux.rocm.gpu.mi300.2) (gh)
'test/inductor/test_perf.py::InplacingTests::test_inplace_custom_op'
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 1, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 10, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 2, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 3, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 4, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 5, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 6, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 7, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 8, 10, linux.s390x) (gh)
Process completed with exit code 1.
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 9, 10, linux.s390x) (gh)
Process completed with exit code 1.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 5, 8, linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 4, 7, linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-cuda12.8-py3.10-gcc11-sm89 / test (default, 4, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (similar failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-py3.13-clang12 / test (dynamo_wrapped, 3, 3, linux.2xlarge) (gh) (disabled by #153707 but the issue was closed recently and a rebase is needed to make it pass)
test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_raises_no_warnings_logging_tensor
pull / linux-jammy-py3.13-clang12 / test (einops, 1, 1, linux.2xlarge) (gh) (similar failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-py3.9-clang12 / test (dynamo_wrapped, 3, 3, linux.2xlarge) (gh) (disabled by #153707 but the issue was closed recently and a rebase is needed to make it pass)
test_autograd.py::TestAutogradFunctional::test_jacobian_vectorize_raises_no_warnings_logging_tensor
pull / linux-jammy-py3.9-clang12 / test (einops, 1, 1, linux.2xlarge) (gh) (similar failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 4, 8, linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (trunk failure)
inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda
periodic / linux-jammy-cuda12.8-py3.10-gcc11 / test (nogpu_AVX512, 3, 3, linux.4xlarge) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
periodic / linux-jammy-cuda12.8-py3.10-gcc11 / test (nogpu_NO_AVX2, 2, 2, linux.4xlarge) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 1, 7, linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (trunk failure)
test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 2, 7, linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (trunk failure)
test_transformers.py::TestSDPAFailureModesCUDA::test_mem_eff_attention_large_seq_len_uniform_attention_cuda
pull / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
test_transformers.py::TestSDPAFailureModesCUDA::test_mem_eff_attention_large_seq_len_uniform_attention_cuda
pull / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 4, 5, linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-cuda12.8-py3.10-gcc11-sm89 / test (default, 3, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
test_transformers.py::TestSDPAFailureModesCUDA::test_mem_eff_attention_large_seq_len_uniform_attention_cuda
pull / linux-jammy-py3.10-clang18-asan / test (default, 3, 6, linux.4xlarge) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-py3.13-clang12 / test (default, 5, 5, linux.4xlarge) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-py3.9-clang12 / test (default, 5, 5, linux.4xlarge) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, linux.2xlarge) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
trunk / macos-py3-arm64 / test (default, 1, 3, macos-m1-stable) (gh) (trunk failure)
dynamo/test_reorder_logs.py::ReorderLogsTests::test_constant_mutation
trunk / win-vs2022-cpu-py3 / build (gh) (trunk failure)
C:\actions-runner\_work\pytorch\pytorch\torch\nativert\executor\memory\GreedyBySize.cpp(119): error C7555: use of designated initializers requires at least '/std:c++20'
trunk / win-vs2022-cuda12.6-py3 / build (gh) (trunk failure)
C:\actions-runner\_work\pytorch\pytorch\torch\nativert\executor\memory\GreedyBySize.cpp(119): error C7555: use of designated initializers requires at least '/std:c++20'

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh) (#153987)
MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

test/run_test.py

jithunnair-amd · 2025-07-05T04:25:56Z

Avg duration of each rocm-mi300 shard went down from 2.1h to 1.5h
Avg duration of each rocm shard went down from 2h to 1.2h

jithunnair-amd · 2025-07-08T06:25:41Z

@pytorchbot merge -f "ROCm jobs ran with TD enabled. Other failures not related to this PR"

pytorchmergebot · 2025-07-08T06:27:19Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Enable target-determination (TD) for ROCm CI

773a2c7

pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Jun 21, 2025

pytorchbot added the open source label Jun 21, 2025

clee2000 reviewed Jun 23, 2025

View reviewed changes

test/run_test.py Show resolved Hide resolved

jithunnair-amd added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 23, 2025

jithunnair-amd added 2 commits June 23, 2025 16:21

Enable TD for any workflows with "rocm" in the name

93a6628

Lint

a66e061

jithunnair-amd added ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 ciflow/periodic-rocm-mi300 Trigger "distributed" config CI on ROCm MI300/MI325 labels Jul 3, 2025

jithunnair-amd added 3 commits July 3, 2025 18:12

Restrict TD to rocm and rocm-mi300 workflows

6166a8a

Lint

8473e73

Lint

4dcc2f8

More lint

e1ed5e2

jithunnair-amd requested review from clee2000 and jeffdaily July 5, 2025 04:28

jeffdaily approved these changes Jul 7, 2025

View reviewed changes

clee2000 approved these changes Jul 7, 2025

View reviewed changes

jithunnair-amd marked this pull request as ready for review July 8, 2025 06:25

jithunnair-amd requested a review from a team as a code owner July 8, 2025 06:25

pytorchmergebot added the merging label Jul 8, 2025

pytorchmergebot closed this in 38757d9 Jul 8, 2025

pytorchmergebot added the Merged label Jul 8, 2025

pytorchmergebot removed the merging label Jul 8, 2025

github-actions bot deleted the enable_td_rocm branch August 8, 2025 02:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable target-determination (TD) for ROCm CI#156545

Enable target-determination (TD) for ROCm CI#156545
jithunnair-amd wants to merge 7 commits intomainfrom
enable_td_rocm

jithunnair-amd commented Jun 21, 2025 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Jun 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

jithunnair-amd commented Jul 5, 2025 •

edited

Loading

Uh oh!

jithunnair-amd commented Jul 8, 2025

Uh oh!

pytorchmergebot commented Jul 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jithunnair-amd commented Jun 21, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156545

❌ 13 New Failures, 23 Unrelated Failures

Uh oh!

Uh oh!

jithunnair-amd commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd commented Jul 8, 2025

Uh oh!

pytorchmergebot commented Jul 8, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jithunnair-amd commented Jun 21, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 21, 2025 •

edited

Loading

jithunnair-amd commented Jul 5, 2025 •

edited

Loading