[CI] Add inductor workflow for rocm by pragupta · Pull Request #110544 · pytorch/pytorch

pragupta · 2023-10-04T19:53:20Z

This PR is to create a separate CI job for inductor UTs on ROCm. You will need to add ciflow/inductor tag on PRs to trigger this job. However, the job will run on its own on any commit merged in main. This job takes around 1.5 hours to run and it is run in parallel to other rocm jobs. It is run only on the MI210 CI runners to ensure maximum inductor functionality is tested.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

pytorch-bot · 2023-10-04T19:53:23Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110544

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (5 Unrelated Failures)

As of commit 1db9e19 with merge base 0249c4a ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
test_ops.py::TestCommonCUDA::test_dtypes_bitwise_and_cuda

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)
test_schema_check.py::TestSchemaCheckModeOpInfoCUDA::test_schema_correctness___rand___cuda_int32
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)
test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_right_shift_cuda
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)
test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_and_cuda_int32
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)
test_meta.py::TestMetaCUDA::test_dispatch_symbolic_meta_outplace___rdiv___cuda_bool

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pragupta · 2023-10-04T19:56:09Z

@pytorchbot label "ciflow/inductor"

pragupta · 2023-10-10T03:20:20Z

@pytorchbot rebase

pytorchmergebot · 2023-10-10T03:22:42Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2023-10-10T03:22:48Z

Successfully rebased add-rocm-inductor-ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout add-rocm-inductor-ci && git pull --rebase)

…107760) This PR adds a skip decorator which will disable tests in CI for ROCm inductor workflow. This new workflow will be coming in via #110544 Pull Request resolved: #107760 Approved by: https://github.com/jataylo, https://github.com/pruthvistony, https://github.com/atalman

jithunnair-amd · 2023-10-18T16:54:38Z

Waiting on #110511 to be merged before enabling this workflow so as to not overburden ROCm CI.

pragupta · 2023-11-16T17:29:08Z

@pytorchbot rebase

pytorchmergebot · 2023-11-16T17:30:48Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2023-11-16T17:30:50Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/110544/head returned non-zero exit code 1

Rebasing (1/5)
Rebasing (2/5)
Rebasing (3/5)
Rebasing (4/5)
Auto-merging .ci/pytorch/test.sh
Auto-merging test/inductor/test_aot_inductor.py
CONFLICT (content): Merge conflict in test/inductor/test_aot_inductor.py
error: could not apply 7be8feb4f7d... Disable cpp version of test_aot_inductor test for rocm
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 7be8feb4f7d... Disable cpp version of test_aot_inductor test for rocm

Raised by https://github.com/pytorch/pytorch/actions/runs/6894323604

pragupta · 2023-11-16T17:43:03Z

Saving the passing CI link before rebasing: https://hud.pytorch.org/pr/110544

jithunnair-amd · 2023-11-16T17:44:15Z

Saving the passing CI link before rebasing: https://hud.pytorch.org/pr/110544

Be aware that the link is dynamic; it will also update when you rebase :) Your best bet is to save the links to the specific jobs that ran eg. https://github.com/pytorch/pytorch/actions/runs/6541944809/job/17764595548.

huydhn · 2023-11-16T18:18:18Z

Please help double check the duration of the new test job too. I'm waiting for the current run to finish https://github.com/pytorch/pytorch/actions/runs/6894570049/job/18757398102 to see how long it takes to finish. More shards might be needed if it takes more than 2 hours

malfet · 2023-11-16T22:53:14Z

.ci/pytorch/test.sh

+#Set Default values for these variables in case they are not set
+SHARD_NUMBER="${SHARD_NUMBER:=1}"
+NUM_TEST_SHARDS="${NUM_TEST_SHARDS:=1}"


Why this is needed? I would prefer it to error out if SHARD_NUMBER is not defined

In order to run inductor config, it seems that SHARD_NUMBER is required now. (https://github.com/pytorch/pytorch/pull/110544/files#diff-9709e5db13aeac90c0312b2b8da34b37cf51242ec789ca959fcf1ba295a8da7aR1101)

Although this variable is always defined in all the CI yaml files that trigger these tests, it's nice to be able to run this file locally outside of CI context.

pragupta · 2023-11-17T20:06:53Z

Please help double check the duration of the new test job too. I'm waiting for the current run to finish https://github.com/pytorch/pytorch/actions/runs/6894570049/job/18757398102 to see how long it takes to finish. More shards might be needed if it takes more than 2 hours

@huydhn -- I have noticed that it takes around 1.5 hours on average for this job to run. Is that sufficient for one shard?

jithunnair-amd · 2024-01-05T17:47:36Z

@pytorchbot rebase

pytorchmergebot · 2024-01-05T17:49:29Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-01-05T17:49:34Z

Successfully rebased add-rocm-inductor-ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout add-rocm-inductor-ci && git pull --rebase)

jithunnair-amd · 2024-01-08T19:29:14Z

@huydhn @malfet Can you please approve and merge this PR?

jeffdaily · 2024-01-08T22:56:40Z

@pytorchbot merge

pytorchmergebot · 2024-01-08T22:58:35Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Oct 4, 2023

pytorch-bot bot added the ciflow/inductor label Oct 4, 2023

pytorchbot added the open source label Oct 4, 2023

pragupta force-pushed the add-rocm-inductor-ci branch from b60aef6 to e8aaa65 Compare October 9, 2023 15:34

pytorchmergebot force-pushed the add-rocm-inductor-ci branch from e8aaa65 to e26d2f7 Compare October 10, 2023 03:22

pragupta mentioned this pull request Oct 12, 2023

Added decorator skipRocmIfTorchInductor and skipped failing tests #107760

Closed

pragupta force-pushed the add-rocm-inductor-ci branch 3 times, most recently from 721b16e to 909146e Compare October 17, 2023 02:56

github-actions bot added the module: inductor label Oct 17, 2023

pragupta force-pushed the add-rocm-inductor-ci branch from 909146e to b69a69c Compare November 16, 2023 17:53

pragupta marked this pull request as ready for review November 16, 2023 17:59

pragupta requested a review from a team as a code owner November 16, 2023 17:59

malfet reviewed Nov 16, 2023

View reviewed changes

janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 17, 2023

pragupta force-pushed the add-rocm-inductor-ci branch from 9e076a3 to 044ecdf Compare December 11, 2023 17:28

jithunnair-amd requested a review from huydhn December 14, 2023 19:52

jithunnair-amd mentioned this pull request Jan 5, 2024

[ROCm] upgrade CI to 6.0 #116270

Closed

jithunnair-amd approved these changes Jan 5, 2024

View reviewed changes

pragupta and others added 10 commits January 5, 2024 17:49

[CI] Add inductor workflow for rocm

ae7c9c0

Update default shard values in test.sh

b53686a

Update test.sh

829f592

Disable cpp version of test_aot_inductor test for rocm

6c9a223

Disable distributed tests

ac8bcf4

Enable distributed UTs

51f1c29

Disable test_aot_inductor cpp test

6d72284

linting

fefb592

Update inductor.yml

0b5cc43

Update inductor.yml

1db9e19

pytorchmergebot force-pushed the add-rocm-inductor-ci branch from 4931a1f to 1db9e19 Compare January 5, 2024 17:49

jithunnair-amd requested a review from jansel January 8, 2024 21:25

jansel approved these changes Jan 8, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 8, 2024

pytorchmergebot added the merging label Jan 8, 2024

huydhn approved these changes Jan 8, 2024

View reviewed changes

pytorchmergebot added the Merged label Jan 9, 2024

pytorchmergebot closed this in 6ca31ae Jan 9, 2024

pytorchmergebot removed the merging label Jan 9, 2024

Conversation

pragupta commented Oct 4, 2023 • edited by jithunnair-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110544

✅ You can merge normally! (5 Unrelated Failures)

Uh oh!

pragupta commented Oct 4, 2023

Uh oh!

pragupta commented Oct 10, 2023

Uh oh!

pytorchmergebot commented Oct 10, 2023

Uh oh!

pytorchmergebot commented Oct 10, 2023

Uh oh!

jithunnair-amd commented Oct 18, 2023

Uh oh!

pragupta commented Nov 16, 2023

Uh oh!

pytorchmergebot commented Nov 16, 2023

Uh oh!

pytorchmergebot commented Nov 16, 2023

Uh oh!

pragupta commented Nov 16, 2023

Uh oh!

jithunnair-amd commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huydhn commented Nov 16, 2023

Uh oh!

malfet Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

pragupta Jan 5, 2024

Choose a reason for hiding this comment

Uh oh!

pragupta commented Nov 17, 2023

Uh oh!

jithunnair-amd commented Jan 5, 2024

Uh oh!

pytorchmergebot commented Jan 5, 2024

Uh oh!

pytorchmergebot commented Jan 5, 2024

Uh oh!

jithunnair-amd commented Jan 8, 2024

Uh oh!

jeffdaily commented Jan 8, 2024

Uh oh!

pytorchmergebot commented Jan 8, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

pragupta commented Oct 4, 2023 •

edited by jithunnair-amd

Loading

pytorch-bot bot commented Oct 4, 2023 •

edited

Loading

jithunnair-amd commented Nov 16, 2023 •

edited

Loading