[Reland] ROCm CI (Infra + Skips) by petrex · Pull Request #1581 · pytorch/ao

petrex · 2025-01-17T18:13:24Z

This PR to skip the unit test failures for ROCm + infra changes to enable ROCm CI.

NOTE:
This PR aims to enable the ROCm CI testing for torchao only for pushes to main branch. The ROCm tests should start showing up here once this PR is merged: https://hud.pytorch.org/hud/pytorch/ao/main/1?per_page=50&name_filter=regression
Torchao PRs can also trigger the ROCm CI runs using the ciflow/rocm PR label (#1749).
Enabling ROCm CI testing on all torchao PRs will be done in a follow-up PR.

This pull request introduces the skip_if_rocm decorator across various test files to skip tests that are not yet supported on ROCm. The changes ensure that tests are conditionally skipped if ROCm is detected, improving the test suite's compatibility with different environments.

Key changes include:

Cherry-pick ROCm CI infra changes from #999

Configure workflow to trigger ROCm CI only for pushes to main branch, OR on PRs with the `ciflow/rocm` label

Introduction of `skip_if_rocm` decorator:

Added skip_if_rocm import in multiple test files to conditionally skip tests not supported on ROCm. (test/dtypes/test_affine_quantized.py, test/dtypes/test_floatx.py, test/float8/test_base.py, test/hqq/test_hqq_affine.py, test/integration/test_integration.py, test/kernel/test_galore_downproj.py, test/prototype/test_awq.py, test/prototype/test_low_bit_optim.py, test/prototype/test_splitk.py, test/quantization/test_galore_quant.py, test/quantization/test_marlin_qqq.py, test/sparsity/test_marlin.py, test/test_ops.py, test/test_s8s4_linear_cutlass.py, torchao/utils.py) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

Application of `skip_if_rocm` decorator:

Applied @skip_if_rocm("ROCm development in progress") to multiple test functions to skip them when running on ROCm. (test/dtypes/test_affine_quantized.py, test/dtypes/test_floatx.py, test/float8/test_base.py, test/hqq/test_hqq_affine.py, test/integration/test_integration.py, test/kernel/test_galore_downproj.py, test/prototype/test_awq.py, test/prototype/test_low_bit_optim.py, test/prototype/test_splitk.py, test/quantization/test_galore_quant.py, test/quantization/test_marlin_qqq.py, test/sparsity/test_marlin.py) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]

Module-level skips for ROCm:

Added module-level skips for ROCm in specific test files to skip all tests within the module if ROCm is detected. (test/test_ops.py, test/test_s8s4_linear_cutlass.py) [1] [2]

pytorch-bot · 2025-01-17T18:13:28Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1581

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 900cf5b with merge base f6f3322 ():

NEW FAILURES - The following jobs have failed:

Run Float8 Tests / test (H100, linux.aws.h100, --pre torch torchvision torchaudio --index-url https://download.pytor... / linux-job (gh)
RuntimeError: Command
Run TorchAO Experimental Tests / test (macos-14) (gh)
[ FAILED ] test_linear_8bit_act_xbit_weight.Kleidi_dotprod_1x4x32_m101xn34xk128xg64

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2025-01-17T18:22:34Z

Warning: Unknown label ciflow/rocm.
Currently recognized labels are

ciflow/benchmark
ciflow/tutorials

Please add the new label to .github/pytorch-probot.yml

andrewor14 · 2025-01-17T19:18:10Z

Thanks @petrex. Would you also like to reopen #999 or should someone else do it?

petrex · 2025-01-17T19:58:03Z

Thanks @andrewor14 I will work with AMD team on that

petrex · 2025-01-21T22:07:54Z

@amdfaa just cherry-pick your infra changes into this PR so we can have a clearer CI signal. Please help review the changes. thx

amdfaa

lgtm from the infra side

msaroufim · 2025-01-22T22:08:06Z

The breakage on cuda doesns't seem related to you, seems like it's this test FAILED test/quantization/test_quant_api.py::TestQuantFlow::test_quantized_tensor_subclass_int8_dyn_quant - torch._inductor.exc.CppCompileError: C++ compile error in which case @jerryzh168 might need to take a look

jithunnair-amd · 2025-01-24T06:13:07Z

@petrex Looks like more fixes/skips are needed: https://github.com/pytorch/ao/actions/runs/12934492712/job/36084046951?pr=1581
=== 117 failed, 1493 passed, 518 skipped, 49 warnings in 3551.22s (0:59:11) ====

petrex · 2025-02-04T00:13:23Z

I see ROCm tests clean : ========= 1480 passed, 500 skipped, 50 warnings in 3848.80s (1:04:08) ==========

but the job later fails with

Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers

lint

.github/workflows/regression_test.yml

jithunnair-amd · 2025-02-20T19:19:16Z

Just to update on the latest status of this PR, we are almost done with enabling/skipping the functionality for the unit tests on ROCm, but we are finalizing the changes to ensure that the ROCm CI runs will run only on push to main branch for now, in light of our limited CI capacity.

In a follow-up PR, we intend to expand torchao ROCm CI testing to torchao PRs as well.

petrex · 2025-02-20T19:43:59Z

@jithunnair-amd maybe lint the changes?

.github/workflows/regression_test.yml

Add a skip decorator for ROCm to prevent test failures during ongoing ROCm enablement

Add ROCm skip decorator to prevent test failures during ongoing ROCm enablement

jithunnair-amd · 2025-02-21T21:30:20Z

@msaroufim @supriyar Can we please get an approval from a torchao maintainer so we can merge this PR when we have a clean signal on ROCm CI (just adding more skips at this point)?

supriyar · 2025-02-21T22:07:57Z

@jithunnair-amd, for sure. @jcaip will be the one reviewing and approving from torchao side.

jithunnair-amd · 2025-02-21T22:41:36Z

@jcaip ROCm nightly CI passed: https://github.com/pytorch/ao/actions/runs/13464088074/job/37625918055?pr=1581

Please approve and merge, before any more unit test failures creep in :)

jcaip

cc @jithunnair-amd LGTM, thanks for the PR!

This PR to skip the unit test failures for ROCm + infra changes to enable ROCm CI. **NOTE:** This PR aims to enable the ROCm CI testing for torchao _only for pushes to main branch_. The ROCm tests should start showing up here once this PR is merged: https://hud.pytorch.org/hud/pytorch/ao/main/1?per_page=50&name_filter=regression Torchao PRs can also trigger the ROCm CI runs using the `ciflow/rocm` PR label (#1749). Enabling ROCm CI testing on *all* torchao PRs will be done in a follow-up PR. This pull request introduces the `skip_if_rocm` decorator across various test files to skip tests that are not yet supported on ROCm. The changes ensure that tests are conditionally skipped if ROCm is detected, improving the test suite's compatibility with different environments. # Key changes include: ### Cherry-pick ROCm CI infra changes from #999 ### Configure workflow to trigger ROCm CI only for pushes to main branch, OR on PRs with the `ciflow/rocm` label ### Introduction of `skip_if_rocm` decorator: * Added `skip_if_rocm` import in multiple test files to conditionally skip tests not supported on ROCm. (`test/dtypes/test_affine_quantized.py`, `test/dtypes/test_floatx.py`, `test/float8/test_base.py`, `test/hqq/test_hqq_affine.py`, `test/integration/test_integration.py`, `test/kernel/test_galore_downproj.py`, `test/prototype/test_awq.py`, `test/prototype/test_low_bit_optim.py`, `test/prototype/test_splitk.py`, `test/quantization/test_galore_quant.py`, `test/quantization/test_marlin_qqq.py`, `test/sparsity/test_marlin.py`, `test/test_ops.py`, `test/test_s8s4_linear_cutlass.py`, `torchao/utils.py`) [[1]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R24) [[2]](diffhunk://#diff-0e811fa3416cd87d9a25b4fb680890098c69aa33ca4db4d347d4a10cc41e0eb3L30-R30) [[3]](diffhunk://#diff-05925b4469eb63ab854cc9891f088f570fa3822cdaeb4de109e0b1b9ab5038a7R21) [[4]](diffhunk://#diff-a9708dc28f15bb9cf665417e6c66601f9e8e2f1f672d1858603b74fa879a3357R13) [[5]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R83) [[6]](diffhunk://#diff-4b0ddf8d1e85f4b4f1067f8d1d3e6b4d48785b3675c7202bf49bfbb1079d682fR14) [[7]](diffhunk://#diff-66249d5a8ed995b0a8e22c6354d6b270c5feeb982cb79a28f7c1b929700e89f4L8-R12) [[8]](diffhunk://#diff-244d33d1e8c30e765556011a4d3b76509f61433a346ba12ffc3115144e895aedR33) [[9]](diffhunk://#diff-2bcf3336ff64bfef786e6126813db46040b93628cab5faff3f0f5ed2cb077bf2L16-R24) [[10]](diffhunk://#diff-51ddab022797064be44ca38c87a56c6e87cd69444f4c6151a11b7f0141aef2b9R21) [[11]](diffhunk://#diff-133d8c7492ee2e7536328c8391545610750774e43d128d258380cb6787bb9e93L22-R22) [[12]](diffhunk://#diff-a58427e02fb5b05d26e03e8c2d216e5ae379d82084fd14bf77ea127b5505a43cL18-R18) [[13]](diffhunk://#diff-d183f2afc51d6a59bc70094e8f476d2468c45e415500f6eb60abad955e065156R22-R24) [[14]](diffhunk://#diff-85cc98d31eb8056e082ebdfbf2979aaa046ffc08bbacd4a65a31795b51998645R10-R12) [[15]](diffhunk://#diff-d2a11602a79e83305208472f1abe6a4106f02ce62a7f9524007181813863fcf6R10) ### Application of `skip_if_rocm` decorator: * Applied `@skip_if_rocm("ROCm development in progress")` to multiple test functions to skip them when running on ROCm. (`test/dtypes/test_affine_quantized.py`, `test/dtypes/test_floatx.py`, `test/float8/test_base.py`, `test/hqq/test_hqq_affine.py`, `test/integration/test_integration.py`, `test/kernel/test_galore_downproj.py`, `test/prototype/test_awq.py`, `test/prototype/test_low_bit_optim.py`, `test/prototype/test_splitk.py`, `test/quantization/test_galore_quant.py`, `test/quantization/test_marlin_qqq.py`, `test/sparsity/test_marlin.py`) [[1]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R93) [[2]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R173) [[3]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R186) [[4]](diffhunk://#diff-0e811fa3416cd87d9a25b4fb680890098c69aa33ca4db4d347d4a10cc41e0eb3R111) [[5]](diffhunk://#diff-05925b4469eb63ab854cc9891f088f570fa3822cdaeb4de109e0b1b9ab5038a7R427) [[6]](diffhunk://#diff-a9708dc28f15bb9cf665417e6c66601f9e8e2f1f672d1858603b74fa879a3357R114) [[7]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R571) [[8]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R690) [[9]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R710) [[10]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R904) [[11]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R924) [[12]](diffhunk://#diff-4b0ddf8d1e85f4b4f1067f8d1d3e6b4d48785b3675c7202bf49bfbb1079d682fR33) [[13]](diffhunk://#diff-66249d5a8ed995b0a8e22c6354d6b270c5feeb982cb79a28f7c1b929700e89f4R120) [[14]](diffhunk://#diff-244d33d1e8c30e765556011a4d3b76509f61433a346ba12ffc3115144e895aedR116) [[15]](diffhunk://#diff-2bcf3336ff64bfef786e6126813db46040b93628cab5faff3f0f5ed2cb077bf2L16-R24) [[16]](diffhunk://#diff-51ddab022797064be44ca38c87a56c6e87cd69444f4c6151a11b7f0141aef2b9R86) [[17]](diffhunk://#diff-133d8c7492ee2e7536328c8391545610750774e43d128d258380cb6787bb9e93R48) [[18]](diffhunk://#diff-133d8c7492ee2e7536328c8391545610750774e43d128d258380cb6787bb9e93R70) [[19]](diffhunk://#diff-a58427e02fb5b05d26e03e8c2d216e5ae379d82084fd14bf77ea127b5505a43cR40) [[20]](diffhunk://#diff-a58427e02fb5b05d26e03e8c2d216e5ae379d82084fd14bf77ea127b5505a43cL51-R58) ### Module-level skips for ROCm: * Added module-level skips for ROCm in specific test files to skip all tests within the module if ROCm is detected. (`test/test_ops.py`, `test/test_s8s4_linear_cutlass.py`) [[1]](diffhunk://#diff-d183f2afc51d6a59bc70094e8f476d2468c45e415500f6eb60abad955e065156R22-R24) [[2]](diffhunk://#diff-85cc98d31eb8056e082ebdfbf2979aaa046ffc08bbacd4a65a31795b51998645R10-R12)

pytorch-bot bot added the device: rocm label Jan 17, 2025

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 17, 2025

petrex force-pushed the skipROCmTest branch from e402a93 to d491087 Compare January 17, 2025 18:16

petrex added the module: not user facing Use this tag if you don't want this PR to show up in release notes label Jan 17, 2025

pytorch-bot bot added the ciflow/rocm label Jan 17, 2025

petrex self-assigned this Jan 17, 2025

petrex requested a review from jainapurva January 17, 2025 18:26

petrex mentioned this pull request Jan 17, 2025

Revert "Skip Unit Tests for ROCm CI" #1580

Merged

petrex requested a review from andrewor14 January 17, 2025 18:27

andrewor14 changed the title ~~Skip ROCm Tests in CI~~ Fix imports after skipping ROCm Tests in CI Jan 17, 2025

andrewor14 changed the title ~~Fix imports after skipping ROCm Tests in CI~~ [Reland] Skip ROCm Tests in CI Jan 17, 2025

petrex force-pushed the skipROCmTest branch from 5104916 to 88448d2 Compare January 21, 2025 22:02

petrex requested a review from amdfaa January 21, 2025 22:08

petrex changed the title ~~[Reland] Skip ROCm Tests in CI~~ [Reland] ROCm CI (Infra + Skips) Jan 21, 2025

petrex requested a review from msaroufim January 22, 2025 19:21

amdfaa reviewed Jan 22, 2025

View reviewed changes

petrex force-pushed the skipROCmTest branch from 26fe1f1 to bb36700 Compare January 23, 2025 07:02

petrex force-pushed the skipROCmTest branch from 4527d3f to bbf74f9 Compare January 24, 2025 17:37

pytorch-bot bot added the ci-no-td label Feb 4, 2025

Peter Y. Yeh and others added 3 commits February 5, 2025 11:24

skip failing unit tests for ROCm CI

afb1d72

update skip_if_rocm import

ec02df5

lint

Enable ROCM in CI

d6ec43f

Peter Y. Yeh and others added 4 commits February 5, 2025 11:42

fix auto-merge

09c0f8c

Update regression_test.yml

387d321

Update regression_test.yml

0b83758

Update regression_test.yml

bef9d17

jithunnair-amd reviewed Feb 20, 2025

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

Update regression_test.yml

fff25bd

jithunnair-amd added 2 commits February 20, 2025 13:31

Merge branch 'main' into skipROCmTest

8426b7a

Update test_ops.py

14bd4cc

jithunnair-amd requested a review from supriyar February 20, 2025 19:39

Attempt to disable only ROCm matrix entries for non-push-to-main

127b445

jithunnair-amd mentioned this pull request Feb 20, 2025

Add ability to enable/disable linux_job_v2 workflow calls pytorch/test-infra#6309

Closed

Attempt to disable only ROCm matrix entries for non-push-to-main - 2

61e86c2

jithunnair-amd force-pushed the skipROCmTest branch from 2da6cd8 to 61e86c2 Compare February 20, 2025 20:24

huydhn reviewed Feb 20, 2025

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

jithunnair-amd force-pushed the skipROCmTest branch from 80711c5 to da9f271 Compare February 21, 2025 14:50

Add new regression_test_rocm.yml as per upstream recommendation

a6958d7

jithunnair-amd force-pushed the skipROCmTest branch from da9f271 to a6958d7 Compare February 21, 2025 14:51

jithunnair-amd and others added 4 commits February 21, 2025 14:59

Ruff fixes

75e0058

Add skip_if_rocm decorator to test_workflow_e2e_numerics

e6ecd1f

Add a skip decorator for ROCm to prevent test failures during ongoing ROCm enablement

lint

27d0d48

Add skip_if_rocm decorator to test_float8_utils

900cf5b

Add ROCm skip decorator to prevent test failures during ongoing ROCm enablement

supriyar requested a review from jcaip February 21, 2025 22:08

jcaip approved these changes Feb 21, 2025

View reviewed changes

jcaip merged commit ed361ff into pytorch:main Feb 21, 2025
16 of 18 checks passed

Conversation

petrex commented Jan 17, 2025 • edited by jithunnair-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key changes include:

Cherry-pick ROCm CI infra changes from #999

Configure workflow to trigger ROCm CI only for pushes to main branch, OR on PRs with the ciflow/rocm label

Introduction of skip_if_rocm decorator:

Application of skip_if_rocm decorator:

Module-level skips for ROCm:

Uh oh!

pytorch-bot bot commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1581

❌ 2 New Failures

Uh oh!

pytorch-bot bot commented Jan 17, 2025

Uh oh!

andrewor14 commented Jan 17, 2025

Uh oh!

petrex commented Jan 17, 2025

Uh oh!

petrex commented Jan 21, 2025

Uh oh!

amdfaa left a comment

Choose a reason for hiding this comment

Uh oh!

msaroufim commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd commented Jan 24, 2025

Uh oh!

petrex commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jithunnair-amd commented Feb 20, 2025

Uh oh!

petrex commented Feb 20, 2025

Uh oh!

Uh oh!

jithunnair-amd commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

supriyar commented Feb 21, 2025

Uh oh!

jithunnair-amd commented Feb 21, 2025

Uh oh!

jcaip left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

petrex commented Jan 17, 2025 •

edited by jithunnair-amd

Loading

Configure workflow to trigger ROCm CI only for pushes to main branch, OR on PRs with the `ciflow/rocm` label

Introduction of `skip_if_rocm` decorator:

Application of `skip_if_rocm` decorator:

pytorch-bot bot commented Jan 17, 2025 •

edited

Loading

msaroufim commented Jan 22, 2025 •

edited

Loading

petrex commented Feb 4, 2025 •

edited

Loading

jithunnair-amd commented Feb 21, 2025 •

edited

Loading

jcaip left a comment •

edited

Loading