Fix compile mode limitation in associative_scan tests by bohnstingl · Pull Request #139929 · pytorch/pytorch

bohnstingl · 2024-11-06T21:52:52Z

Fix compile mode limitation in associative_scan tests where by mistake only the compile_mode='fake' was used

cc @ydwu4

pytorch-bot · 2024-11-06T21:52:56Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139929

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 565a794 with merge base 2f3a5a1 ():

NEW FAILURES - The following jobs have failed:

inductor-cu124-unittest / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float16
inductor-cu124-unittest / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cummax_cummin_cuda
inductor-cu124-unittest / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
inductor-cu124-unittest / cuda12.4-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float16
inductor-cu124-unittest / cuda12.4-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cummax_cummin_cuda
inductor-cu124-unittest / cuda12.4-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
inductor-micro-benchmark / cuda12.1-py3.10-gcc9-sm80 / test (inductor-micro-benchmark, 1, 1, awsa100.linux.gcp.a100) (gh)
RuntimeError: Error: accessing tensor output of CUDAGraphs that has been overwritten by a subsequent run. Stack trace: File "/var/lib/jenkins/workspace/benchmarks/gpt_fast/generate.py", line 93, in decode_one_token
inductor-unittest / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float16
inductor-unittest / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cummax_cummin_cuda
inductor-unittest / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
inductor-unittest / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float16
inductor-unittest / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check) (gh)
test_torch.py::TestTorchDeviceTypeCUDA::test_cummax_cummin_cuda
inductor-unittest / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, lf.linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, lf.linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
rocm / linux-focal-rocm6.2-py3.10 / test (default, 6, 6, linux.rocm.gpu.2, rerun_disabled_tests) (gh)
'Test'
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 4, 5, lf.linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 5, 5, lf.linux.g5.4xlarge.nvidia.gpu, rerun_disabled_tests) (gh)
'Test'

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, lf.linux.2xlarge, rerun_disabled_tests) (gh) (detected as infra flaky with no log or failing log classifier)
pull / linux-focal-py3.11-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge, rerun_disabled_tests) (gh) (detected as infra flaky with no log or failing log classifier)
pull / linux-focal-py3.12-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge, rerun_disabled_tests) (gh) (detected as infra flaky with no log or failing log classifier)
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, lf.linux.2xlarge, rerun_disabled_tests) (gh) (detected as infra flaky with no log or failing log classifier)
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge, rerun_disabled_tests) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / macos-py3-arm64 / test (default, 1, 3, macos-m1-stable, rerun_disabled_tests) (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) (gh) (trunk failure)
distributed/test_c10d_ucc.py::DistributedDataParallelTest::test_save_load_checkpoint

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

unstable / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, lf.linux.2xlarge) (gh)
kernels/quantized/test/test_out_variants.py::TestOutVariants::test_quantize_per_tensor_to_out_variant

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ydwu4 · 2024-11-07T03:09:28Z

@pytorchbot merge

pytorch-bot · 2024-11-07T03:09:32Z

This PR needs to be approved by an authorized maintainer before merge.

ydwu4 · 2024-11-08T18:32:06Z

@pytorchbot merge

pytorchmergebot · 2024-11-08T18:36:49Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

ZainRizvi · 2024-11-08T21:25:11Z

@pytorchbot revert -c nosignal -m "This test is breaking in trunk somehow, which is really weird. Might be a land race. functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda GH job link HUD commit link"

pytorchmergebot · 2024-11-08T21:26:35Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2024-11-08T21:26:43Z

@bohnstingl your PR has been successfully reverted.

This reverts commit 7fa94f0. Reverted #139929 on behalf of https://github.com/ZainRizvi due to This test is breaking in trunk somehow, which is really weird. functorch/test_control_flow.py::AssociativeScanTests::test_associative_scan_binary_operator_compile_mode_compile_dynamic_shape_combine_mode_pointwise_reverse_False_cuda [GH job link](https://github.com/pytorch/pytorch/actions/runs/11747748990/job/32732254909) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/7fa94f03635709a30ef85c6955dcdd5051e72e71) ([comment](#139929 (comment)))

This PR was reopened (likely due to being reverted), so your approval was removed. Please request another review.

malfet · 2024-11-09T02:29:06Z

This test is breaking in trunk somehow, which is really weird.

Nothing weird about it, PR description is a bit misleading, it's not a typo, though it restrict test functionality a lot.

bohnstingl · 2024-11-09T06:26:23Z

Hi @malfet,
Thank you for bringing this up. I have tested this PR locally and for me it was working and also the CI seem to work. I will test it again.

I have also updated the description and the title to avoid confusion.

pytorch-bot bot added the topic: not user facing topic category label Nov 6, 2024

pytorchbot added the open source label Nov 6, 2024

ydwu4 added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 6, 2024

ydwu4 previously approved these changes Nov 7, 2024

View reviewed changes

pytorchmergebot added the merging label Nov 8, 2024

pytorchmergebot closed this in 7fa94f0 Nov 8, 2024

pytorchmergebot added Merged and removed merging labels Nov 8, 2024

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Nov 8, 2024

pytorchmergebot reopened this Nov 8, 2024

malfet added the ciflow/inductor label Nov 9, 2024

bohnstingl changed the title ~~Fix typo in associative_scan tests~~ Fix compile mode limitation in associative_scan tests Nov 9, 2024

bohnstingl closed this Nov 11, 2024

bohnstingl force-pushed the fix_compile_mode_test_assoc_scan branch from 82e87e8 to 565a794 Compare November 11, 2024 13:29

huydhn temporarily deployed to scribe-protected November 11, 2024 13:30 — with GitHub Actions Inactive

pytorchmergebot temporarily deployed to scribe-protected November 11, 2024 13:38 — with GitHub Actions Inactive

pytorchmergebot temporarily deployed to scribe-protected November 11, 2024 13:39 — with GitHub Actions Inactive

pytorchmergebot temporarily deployed to scribe-protected November 11, 2024 13:40 — with GitHub Actions Inactive

iremyux temporarily deployed to upload-stats November 11, 2024 13:40 — with GitHub Actions Inactive

pytorchmergebot temporarily deployed to upload-stats November 11, 2024 13:41 — with GitHub Actions Inactive

pytorch-bot bot temporarily deployed to upload-stats November 11, 2024 13:41 Inactive

pytorchmergebot temporarily deployed to scribe-protected November 11, 2024 13:41 — with GitHub Actions Inactive

pytorchmergebot temporarily deployed to scribe-protected November 11, 2024 13:42 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix compile mode limitation in associative_scan tests#139929

Fix compile mode limitation in associative_scan tests#139929
bohnstingl wants to merge 0 commit intopytorch:mainfrom
bohnstingl:fix_compile_mode_test_assoc_scan

bohnstingl commented Nov 6, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Nov 6, 2024 •

edited

Loading

Uh oh!

ydwu4 commented Nov 7, 2024

Uh oh!

pytorch-bot bot commented Nov 7, 2024

Uh oh!

ydwu4 commented Nov 8, 2024

Uh oh!

pytorchmergebot commented Nov 8, 2024

Uh oh!

ZainRizvi commented Nov 8, 2024 •

edited

Loading

Uh oh!

pytorchmergebot commented Nov 8, 2024

Uh oh!

pytorchmergebot commented Nov 8, 2024

Uh oh!

malfet commented Nov 9, 2024

Uh oh!

bohnstingl commented Nov 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Conversation

bohnstingl commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139929

❌ 20 New Failures, 8 Unrelated Failures

Uh oh!

ydwu4 commented Nov 7, 2024

Uh oh!

pytorch-bot bot commented Nov 7, 2024

Uh oh!

ydwu4 commented Nov 8, 2024

Uh oh!

pytorchmergebot commented Nov 8, 2024

Merge started

Uh oh!

ZainRizvi commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorchmergebot commented Nov 8, 2024

Uh oh!

pytorchmergebot commented Nov 8, 2024

Uh oh!

malfet commented Nov 9, 2024

Uh oh!

bohnstingl commented Nov 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

bohnstingl commented Nov 6, 2024 •

edited

Loading

pytorch-bot bot commented Nov 6, 2024 •

edited

Loading

ZainRizvi commented Nov 8, 2024 •

edited

Loading