[ROCm] Update setup-rocm for almalinux-based images by amdfaa · Pull Request #143590 · pytorch/pytorch

amdfaa · 2024-12-19T18:58:17Z

Needed for pytorch/test-infra#6104 and pytorch/ao#999

Explicitly specify repo and branch in pytorch/pytorch/.github/actions/diskspace-cleanup@main to be able to use setup-rocm in test-infra's .github/workflows/linux_job_v2.yml (like in PR Enable linux_job_v2.yml workflow for ROCm test-infra#6104), otherwise Github Actions complains about not finding diskspace-cleanup action in test-infra repo.
Use RUNNER_TEMP instead of /tmp
Add bin group permissions for Almalinux images due to difference in default OS group numbering in Ubuntu vs Almalinux

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2024-12-19T18:58:21Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143590

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 22 New Failures, 1 Cancelled Job

As of commit 95c5aa6 with merge base 7ced49d ():

NEW FAILURES - The following jobs have failed:

Lint / lintrunner-noclang / linux-job (gh)
>>> Lint for test/test_transformers.py:
linux-binary-manywheel / manywheel-py3_9-cuda12_6-test / test (gh)
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 5, 1) but found runtime version (9, 1, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN. one possibility is that there is a conflicting cuDNN in LD_LIBRARY_PATH.
pull / cuda12.4-py3.10-gcc9-sm75 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-cuda11.8-py3.10-gcc9 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-cuda12.4-py3.10-gcc9 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-py3_9-clang9-xla / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-py3.13-clang10 / build (gh)
Final attempt failed. Child_process exited with error code 1
pull / linux-focal-py3.9-clang10 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-py3.9-clang10-onnx / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-focal-rocm6.2-py3.10 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-cuda11.8-cudnn9-py3.9-clang12 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3-clang12-executorch / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3-clang12-mobile-build / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3.10-clang15-asan / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3.9-gcc11 / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3.9-gcc11-mobile-lightweight-dispatch-build / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3.9-gcc11-no-ops / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / linux-jammy-py3.9-gcc11-pch / build (gh)
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/actions/upload-sccache-stats'. Did you forget to run actions/checkout before running your local action?
pull / win-vs2019-cpu-py3 / build (gh)
AttributeError: module 'distutils' has no attribute '_msvccompiler'
trunk / win-vs2019-cpu-py3 / build (gh)
AttributeError: module 'distutils' has no attribute '_msvccompiler'
trunk / win-vs2019-cuda12.1-py3 / build (gh)
AttributeError: module 'distutils' has no attribute '_msvccompiler'

CANCELLED JOB - The following job was cancelled. Please retry:

trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / build (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jithunnair-amd · 2024-12-19T19:51:57Z

@huydhn This is one of 3 PRs to get torchao CI working on ROCm. Can you please review and approve the PR?

amdfaa · 2024-12-19T19:52:09Z

.github/actions/setup-rocm/action.yml

      run: |
        # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
-        echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}"
+        echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon --group-add bin" >> "${GITHUB_ENV}"


The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively. This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal. This video group ID maps to subgid 1 inside the docker image. The group name corresponding to group
ID 1 can change depending on the OS, so both are necessary.

atalman

lgtm, please fix lint before merging. Looks like lint failure is legit

jithunnair-amd · 2024-12-20T22:47:14Z

trunk workflow ROCm jobs passed, so these changes seem to have not broken existing workflows.

@pytorchbot merge -f "Current lint and other CI failures are unrelated to this PR, probably due to older base. ROCm CI succeeded."

pytorchmergebot · 2024-12-20T22:48:42Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Relates to pytorch/ao#999 Needs pytorch/pytorch#143590

amdfaa added 2 commits November 19, 2024 17:10

change reference

bd44f9d

Update action.yml

4d16297

pytorch-bot bot added the topic: not user facing topic category label Dec 19, 2024

facebook-github-bot added the module: rocm AMD GPU support for Pytorch label Dec 19, 2024

pytorchbot added the open source label Dec 19, 2024

jithunnair-amd changed the title ~~Torchao experiment~~ [ROCm] Update setup-rocm for almalinux-based images Dec 19, 2024

jithunnair-amd requested a review from huydhn December 19, 2024 19:51

jithunnair-amd marked this pull request as ready for review December 19, 2024 19:51

jithunnair-amd requested a review from a team as a code owner December 19, 2024 19:51

amdfaa commented Dec 19, 2024

View reviewed changes

amdfaa and others added 2 commits December 19, 2024 13:57

Update action.yml

227aa9e

Update action.yml

623993c

jithunnair-amd mentioned this pull request Dec 19, 2024

Enable linux_job.yml workflow for ROCm pytorch/test-infra#6003

Closed

atalman approved these changes Dec 19, 2024

View reviewed changes

amdfaa added 2 commits December 19, 2024 16:41

Update action.yml

4affcc8

Update action.yml

e9d0f07

jithunnair-amd added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 19, 2024

Update _rocm-test.yml

95c5aa6

pytorchmergebot added the merging label Dec 20, 2024

pytorchmergebot added the Merged label Dec 20, 2024

pytorchmergebot closed this in a3b04d4 Dec 20, 2024

pytorchmergebot removed the merging label Dec 20, 2024

pytorch-bot bot added the ciflow/rocm Trigger "default" config CI on ROCm label Dec 23, 2024

jithunnair-amd mentioned this pull request Jan 8, 2025

Enable linux_job_v2.yml workflow for ROCm pytorch/test-infra#6104

Merged

amdfaa added a commit to pytorch/test-infra that referenced this pull request Jan 16, 2025

Enable linux_job_v2.yml workflow for ROCm (#6104)

15110a3

Relates to pytorch/ao#999 Needs pytorch/pytorch#143590

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ROCm] Update setup-rocm for almalinux-based images#143590

[ROCm] Update setup-rocm for almalinux-based images#143590
amdfaa wants to merge 7 commits intopytorch:mainfrom
amdfaa:torchao_experiment

amdfaa commented Dec 19, 2024 •

edited by jithunnair-amd

Loading

Uh oh!

pytorch-bot bot commented Dec 19, 2024 •

edited

Loading

Uh oh!

jithunnair-amd commented Dec 19, 2024

Uh oh!

amdfaa Dec 19, 2024 •

edited by jithunnair-amd

Loading

Uh oh!

atalman left a comment

Uh oh!

jithunnair-amd commented Dec 20, 2024

Uh oh!

pytorchmergebot commented Dec 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

amdfaa commented Dec 19, 2024 • edited by jithunnair-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Dec 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143590

❌ 22 New Failures, 1 Cancelled Job

Uh oh!

jithunnair-amd commented Dec 19, 2024

Uh oh!

amdfaa Dec 19, 2024 • edited by jithunnair-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd commented Dec 20, 2024

Uh oh!

pytorchmergebot commented Dec 20, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

amdfaa commented Dec 19, 2024 •

edited by jithunnair-amd

Loading

pytorch-bot bot commented Dec 19, 2024 •

edited

Loading

amdfaa Dec 19, 2024 •

edited by jithunnair-amd

Loading