[ROCm] Skip pointwise associative scan tests due to regression by jataylo · Pull Request #135995 · pytorch/pytorch

jataylo · 2024-09-13T13:20:51Z

#133012 caused a regression on ROCm causing pointwise scan tests to fail

ERROR: test_pointwise_associative_scan_tuple_reverse_True_combine_mode_pointwise_cuda
ERROR: test_pointwise_associative_scan_tuple_reverse_False_combine_mode_pointwise_cuda
ERROR: test_pointwise_associative_scan_complex_pytree_reverse_True_combine_mode_pointwise_cuda
ERROR: test_pointwise_associative_scan_complex_pytree_reverse_False_combine_mode_pointwise_cuda 
ERROR: test_pointwise_associative_scan_binary_operator_reverse_True_combine_mode_pointwise_cuda 
ERROR: test_pointwise_associative_scan_binary_operator_reverse_False_combine_mode_pointwise_cuda

Skipping temporarily while triage is underway.

Full log: https://ossci-raw-job-status.s3.amazonaws.com/log/30067645445

  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/graph.py", line 1020, in call_function
    out = lowerings[target](*args, **kwargs)  # type: ignore[index]
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/lowering.py", line 363, in wrapped
    out = decomp_fn(*args, **kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/lowering.py", line 6245, in associative_scan
    raise RuntimeError("Unable to generate code for associative_scan op")
torch._inductor.exc.LoweringException: RuntimeError: Unable to generate code for associative_scan op

NOTE: even "eager" backend fails

  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_higher_order_ops/associative_scan.py", line 338, in associative_scan_op_dense
    raise NotImplementedError("associative_scan is not implemented for eager")
NotImplementedError: associative_scan is not implemented for eager

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @hongxiayang

pytorch-bot · 2024-09-13T13:20:55Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135995

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit fa7b212 with merge base 7ed0563 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 2, 3, linux.2xlarge) (gh) (disabled by #132861, #132862, #134600)
test_transformers.py::TestSDPAPrivateUse1Only::test_fused_sdp_choice_privateuseone

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jithunnair-amd · 2024-09-13T14:30:25Z

@malfet Another PR to skip unit tests and get ROCm CI signal to green while we investigate fix in parallel.

test/functorch/test_control_flow.py

jithunnair-amd · 2024-09-14T05:38:27Z

@pytorchbot merge -f "Skipping these tests gets rocm workflow signal to green. Discussions ongoing on proper fix in parallel"

pytorchmergebot · 2024-09-14T05:39:58Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ch#135995) pytorch#133012 caused a regression on ROCm causing pointwise scan tests to fail ``` ERROR: test_pointwise_associative_scan_tuple_reverse_True_combine_mode_pointwise_cuda ERROR: test_pointwise_associative_scan_tuple_reverse_False_combine_mode_pointwise_cuda ERROR: test_pointwise_associative_scan_complex_pytree_reverse_True_combine_mode_pointwise_cuda ERROR: test_pointwise_associative_scan_complex_pytree_reverse_False_combine_mode_pointwise_cuda ERROR: test_pointwise_associative_scan_binary_operator_reverse_True_combine_mode_pointwise_cuda ERROR: test_pointwise_associative_scan_binary_operator_reverse_False_combine_mode_pointwise_cuda ``` Skipping temporarily while triage is underway. Full log: https://ossci-raw-job-status.s3.amazonaws.com/log/30067645445 ``` File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/graph.py", line 1020, in call_function out = lowerings[target](*args, **kwargs) # type: ignore[index] File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/lowering.py", line 363, in wrapped out = decomp_fn(*args, **kwargs) File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/lowering.py", line 6245, in associative_scan raise RuntimeError("Unable to generate code for associative_scan op") torch._inductor.exc.LoweringException: RuntimeError: Unable to generate code for associative_scan op ``` NOTE: even "eager" backend fails ``` File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_higher_order_ops/associative_scan.py", line 338, in associative_scan_op_dense raise NotImplementedError("associative_scan is not implemented for eager") NotImplementedError: associative_scan is not implemented for eager ``` Pull Request resolved: pytorch#135995 Approved by: https://github.com/malfet

jithunnair-amd · 2024-09-24T18:29:28Z

@pytorchbot cherry-pick --onto release/2.5 -c critical

pytorchbot · 2024-09-24T18:33:50Z

Cherry picking #135995

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x b9b6094793439a8e2132d545ba9760733b02bc7c returned non-zero exit code 1

Auto-merging test/functorch/test_control_flow.py
CONFLICT (content): Merge conflict in test/functorch/test_control_flow.py
error: could not apply b9b6094793... [ROCm] Skip pointwise associative scan tests due to regression (#135995)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

…d on PR #135995)

jithunnair-amd · 2024-09-24T21:29:03Z

The cherry-pick PR is at #136557

* [ROCm] skip test_fp8_cast_and_t on non-MI300 machines (#135917) Fixes #ISSUE_NUMBER Pull Request resolved: #135917 Approved by: https://github.com/malfet (cherry picked from commit 6cdc70b) * Skip pointwise associative scan tests due to regression (changes based on PR #135995) * Cherry-pick fix from #135702 --------- Co-authored-by: Prachi Gupta <prachi.gupta@amd.com> Co-authored-by: Jithun Nair <jithun.nair@amd.com>

[ROCm] Skip pointwise associative scan tests

e585f6c

jataylo requested review from Chillee, jansel, nmacchioni, peterbell10 and ydwu4 September 13, 2024 13:20

pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Sep 13, 2024

pytorchbot added the open source label Sep 13, 2024

jataylo added the ciflow/inductor-rocm Trigger "inductor" config CI on ROCm label Sep 13, 2024

jataylo requested a review from jithunnair-amd September 13, 2024 13:39

jithunnair-amd requested a review from malfet September 13, 2024 14:29

malfet approved these changes Sep 13, 2024

View reviewed changes

test/functorch/test_control_flow.py Show resolved Hide resolved

Lint

fa7b212

pytorchmergebot added the merging label Sep 14, 2024

pytorchmergebot added the Merged label Sep 14, 2024

pytorchmergebot closed this in b9b6094 Sep 14, 2024

pytorchmergebot removed the merging label Sep 14, 2024

jithunnair-amd added a commit that referenced this pull request Sep 24, 2024

Skip pointwise associative scan tests due to regression (changes base…

8f94eaa

…d on PR #135995)

jithunnair-amd mentioned this pull request Sep 24, 2024

[ROCm] Cherry-pick unit test fixes to release/2.5 #136557

Merged

jithunnair-amd mentioned this pull request Sep 24, 2024

[v.2.5.0] Release Tracker #135522

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ROCm] Skip pointwise associative scan tests due to regression#135995

[ROCm] Skip pointwise associative scan tests due to regression#135995
jataylo wants to merge 2 commits intopytorch:mainfrom
jataylo:skip_pointwise_rocm

jataylo commented Sep 13, 2024 •

edited by jithunnair-amd

Loading

Uh oh!

pytorch-bot bot commented Sep 13, 2024 •

edited

Loading

Uh oh!

jithunnair-amd commented Sep 13, 2024

Uh oh!

Uh oh!

jithunnair-amd commented Sep 14, 2024

Uh oh!

pytorchmergebot commented Sep 14, 2024

Uh oh!

jithunnair-amd commented Sep 24, 2024

Uh oh!

pytorchbot commented Sep 24, 2024

Uh oh!

jithunnair-amd commented Sep 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

jataylo commented Sep 13, 2024 • edited by jithunnair-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135995

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

jithunnair-amd commented Sep 13, 2024

Uh oh!

Uh oh!

jithunnair-amd commented Sep 14, 2024

Uh oh!

pytorchmergebot commented Sep 14, 2024

Merge started

Uh oh!

jithunnair-amd commented Sep 24, 2024

Uh oh!

pytorchbot commented Sep 24, 2024

Cherry picking #135995

Uh oh!

jithunnair-amd commented Sep 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jataylo commented Sep 13, 2024 •

edited by jithunnair-amd

Loading

pytorch-bot bot commented Sep 13, 2024 •

edited

Loading