[ROCm] upgrade ROCm CI builds to py3.10 by jataylo · Pull Request #134108 · pytorch/pytorch

jataylo · 2024-08-21T14:36:02Z

Upgrade ROCm CI builds to py3.10

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @hongxiayang

pytorch-bot · 2024-08-21T14:36:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134108

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 9e85c89 with merge base c64ae60 ():

NEW FAILURES - The following jobs have failed:

periodic / ios-build-test / build (default, 1, 1, macos-14-xlarge, SIMULATOR, arm64, 1, 0, 1) (gh)
periodic / linux-focal-rocm6.1-py3.10 / test (distributed, 3, 3, linux.rocm.gpu) (gh)
'Test'

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

periodic / parallelnative-linux-jammy-py3.9-gcc11 / test (default, 1, 3, lf.linux.2xlarge) (gh) (similar failure)
'Test'
pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh) (disabled by #132861, #132862, #134600)
test_transformers.py::TestSDPAPrivateUse1Only::test_fused_sdp_choice_privateuseone
pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, lf.linux.2xlarge) (gh) (disabled by #134602)
test_transformers.py::TestSDPAPrivateUse1Only::test_scaled_dot_product_fused_attention_overrideable_backward

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jataylo · 2024-08-22T09:13:00Z

I don't believe any of these failures are new and the b2b-gemm tests are passing:

2024-08-21T21:58:17.8609540Z Running inductor/test_b2b_gemm 1/1 ... [2024-08-21 21:58:17.860331]
2024-08-21T21:58:17.8610134Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-08-21T21:58:17.8615244Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_b2b_gemm.py', '-m', 'serial', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-21 21:58:17.860949]
2024-08-21T21:58:21.6348271Z 
2024-08-21T21:58:21.6350603Z inductor/test_b2b_gemm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_b2b_gemm_1.1_56d822e46ac72b2e_.log
2024-08-21T21:58:21.6352427Z Running 0 items in this shard:

jataylo · 2024-08-22T15:55:52Z

@pytorchbot rebase

pytorchmergebot · 2024-08-22T15:57:15Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-08-22T15:57:18Z

Successfully rebased rocm-py310-upgrade onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm-py310-upgrade && git pull --rebase)

jataylo · 2024-08-23T13:10:03Z

@malfet could you take a look at this. Should resolve these b2b-gemm issues we pointed out related to py3.8

ROCm failures are expected, and the timeout is due to keep-going enable with the current test_transformers failures.

pruthvistony · 2024-08-26T20:50:37Z

Waiting for this PR #132895 to merge before this PR.

jithunnair-amd · 2024-08-27T22:30:13Z

@pytorchbot rebase

pytorchmergebot · 2024-08-27T22:31:44Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-08-27T22:31:47Z

Successfully rebased rocm-py310-upgrade onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm-py310-upgrade && git pull --rebase)

jataylo · 2024-09-12T15:12:36Z

@pytorchbot rebase

pytorchmergebot · 2024-09-12T15:14:37Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-09-12T15:14:39Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/134108/head returned non-zero exit code 1

Rebasing (1/3)
Auto-merging .ci/docker/build.sh
Auto-merging .github/workflows/inductor-rocm.yml
CONFLICT (content): Merge conflict in .github/workflows/inductor-rocm.yml
Auto-merging .github/workflows/periodic.yml
CONFLICT (content): Merge conflict in .github/workflows/periodic.yml
Auto-merging .github/workflows/pull.yml
CONFLICT (content): Merge conflict in .github/workflows/pull.yml
Auto-merging .github/workflows/rocm.yml
Auto-merging .github/workflows/slow.yml
CONFLICT (content): Merge conflict in .github/workflows/slow.yml
Auto-merging .github/workflows/trunk.yml
CONFLICT (content): Merge conflict in .github/workflows/trunk.yml
error: could not apply 6a0bdb730d... [ROCm] upgrade ROCm CI builds to py3.10
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 6a0bdb730d... [ROCm] upgrade ROCm CI builds to py3.10

Raised by https://github.com/pytorch/pytorch/actions/runs/10833692309

jataylo · 2024-09-16T09:03:53Z

@pytorchbot rebase

pytorchmergebot · 2024-09-16T09:05:22Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-09-16T09:05:25Z

Successfully rebased rocm-py310-upgrade onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm-py310-upgrade && git pull --rebase)

jataylo · 2024-09-16T16:26:25Z

Failures seem unrelated. Opening for review

jataylo · 2024-09-16T16:26:39Z

cc: @malfet

atalman

lgtm. Please followup on periodic / linux-focal-rocm6.1-py3.10 / test (distributed, 3, 3, linux.rocm.gpu)

jithunnair-amd · 2024-09-18T17:37:21Z

lgtm. Please followup on periodic / linux-focal-rocm6.1-py3.10 / test (distributed, 3, 3, linux.rocm.gpu)

@atalman Yes, we're looking into the timeouts seen in the distributed job. However, it is not an issue related to the python version, so it shouldn't be a blocker for this PR.

jithunnair-amd · 2024-09-18T17:37:50Z

@pytorchbot merge -f "CI failures are unrelated to py3.10 upgrade"

pytorchmergebot · 2024-09-18T17:39:21Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Upgrade ROCm CI builds to py3.10 Pull Request resolved: pytorch#134108 Approved by: https://github.com/jeffdaily, https://github.com/jithunnair-amd, https://github.com/atalman

kit1980 · 2024-09-25T23:47:35Z

@pytorchmergebot cherry-pick --onto release/2.5 -c critical

pytorchbot · 2024-09-25T23:51:50Z

Cherry picking #134108

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x bad69044d87b67a8b64691e4e5a7a68147185aa0 returned non-zero exit code 1

Auto-merging .ci/docker/build.sh
Auto-merging .github/workflows/inductor-rocm.yml
CONFLICT (content): Merge conflict in .github/workflows/inductor-rocm.yml
Auto-merging .github/workflows/periodic.yml
CONFLICT (content): Merge conflict in .github/workflows/periodic.yml
Auto-merging .github/workflows/pull.yml
CONFLICT (content): Merge conflict in .github/workflows/pull.yml
Auto-merging .github/workflows/rocm.yml
CONFLICT (content): Merge conflict in .github/workflows/rocm.yml
Auto-merging .github/workflows/slow.yml
CONFLICT (content): Merge conflict in .github/workflows/slow.yml
Auto-merging .github/workflows/trunk.yml
CONFLICT (content): Merge conflict in .github/workflows/trunk.yml
error: could not apply bad69044d8... [ROCm] upgrade ROCm CI builds to py3.10 (#134108)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Upgrade ROCm CI builds to py3.10 Pull Request resolved: #134108 Approved by: https://github.com/jeffdaily, https://github.com/jithunnair-amd, https://github.com/atalman Co-authored-by: Jack Taylor <jack.taylor@amd.com>

jataylo added rocm This tag is for PRs from ROCm team ciflow/rocm Trigger "default" config CI on ROCm ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Aug 21, 2024

jataylo requested review from jeffdaily, jithunnair-amd and pruthvistony August 21, 2024 14:36

pytorch-bot bot added module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Aug 21, 2024

pytorchbot added the open source label Aug 21, 2024

jataylo mentioned this pull request Aug 21, 2024

Make inductor kernel metadata comments more descriptive #126698

Closed

jeffdaily approved these changes Aug 21, 2024

View reviewed changes

jataylo requested a review from malfet August 22, 2024 09:10

jataylo marked this pull request as ready for review August 22, 2024 09:10

jataylo requested a review from a team as a code owner August 22, 2024 09:10

jataylo added rocm priority high priority ROCm PRs from performance or other aspects keep-going Don't stop on first failure, keep running tests until the end labels Aug 22, 2024

jataylo added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Aug 22, 2024

pytorchmergebot force-pushed the rocm-py310-upgrade branch from d24c4dd to 0c28943 Compare August 22, 2024 15:57

pruthvistony marked this pull request as draft August 26, 2024 20:50

pytorchmergebot force-pushed the rocm-py310-upgrade branch from 0c28943 to 6a0bdb7 Compare August 27, 2024 22:31

jataylo force-pushed the rocm-py310-upgrade branch from 196041b to 6988702 Compare September 13, 2024 09:39

[ROCm] upgrade ROCm CI builds to py3.10

9e85c89

pytorchmergebot force-pushed the rocm-py310-upgrade branch from 6988702 to 9e85c89 Compare September 16, 2024 09:05

jataylo marked this pull request as ready for review September 16, 2024 16:26

jithunnair-amd mentioned this pull request Sep 16, 2024

DISABLED test_pcontext_wait_on_a_child_thread (__main__.StartProcessesAsFuncTest) #136164

Closed

jithunnair-amd approved these changes Sep 17, 2024

View reviewed changes

jithunnair-amd requested a review from atalman September 17, 2024 22:13

atalman approved these changes Sep 17, 2024

View reviewed changes

pytorchmergebot added the merging label Sep 18, 2024

pytorchmergebot closed this in bad6904 Sep 18, 2024

pytorchmergebot added Merged and removed merging labels Sep 18, 2024

This was referenced Sep 26, 2024

[ROCm] upgrade ROCm CI builds to py3.10 (#134108) #136696

Merged

[v.2.5.0] Release Tracker #135522

Closed

Conversation

jataylo commented Aug 21, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134108

❌ 2 New Failures, 3 Unrelated Failures

Uh oh!

jataylo commented Aug 22, 2024

Uh oh!

jataylo commented Aug 22, 2024

Uh oh!

pytorchmergebot commented Aug 22, 2024

Uh oh!

pytorchmergebot commented Aug 22, 2024

Uh oh!

jataylo commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pruthvistony commented Aug 26, 2024

Uh oh!

jithunnair-amd commented Aug 27, 2024

Uh oh!

pytorchmergebot commented Aug 27, 2024

Uh oh!

pytorchmergebot commented Aug 27, 2024

Uh oh!

jataylo commented Sep 12, 2024

Uh oh!

pytorchmergebot commented Sep 12, 2024

Uh oh!

pytorchmergebot commented Sep 12, 2024

Uh oh!

jataylo commented Sep 16, 2024

Uh oh!

pytorchmergebot commented Sep 16, 2024

Uh oh!

pytorchmergebot commented Sep 16, 2024

Uh oh!

jataylo commented Sep 16, 2024

Uh oh!

jataylo commented Sep 16, 2024

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd commented Sep 18, 2024

Uh oh!

jithunnair-amd commented Sep 18, 2024

Uh oh!

pytorchmergebot commented Sep 18, 2024

Merge started

Uh oh!

kit1980 commented Sep 25, 2024

Uh oh!

pytorchbot commented Sep 25, 2024

Cherry picking #134108

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jataylo commented Aug 21, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Aug 21, 2024 •

edited

Loading

jataylo commented Aug 23, 2024 •

edited

Loading