handle multiple reductions in node splits & read/write normalization by eellison · Pull Request #168013 · pytorch/pytorch

eellison · 2025-11-17T17:28:56Z

Stack from ghstack (oldest at bottom):

-> handle multiple reductions in node splits & read/write normalization #168013

Another partial fix to #166653:

We had not yet handled multiple reduction vars in tiling splits, which led to the coalesce analysis not seeing the vars as coalesced. See, updated P2043574063, which shows the kernel with correct coalescing by rblock (private link bc private model).

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo @chenyang78

[ghstack-poisoned]

ghstack-source-id: b36f98f Pull Request resolved: #168013

pytorch-bot · 2025-11-17T17:29:02Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/168013

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (9 Unrelated Failures)

As of commit 9a2a344 with merge base 015826f ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

trunk / linux-jammy-rocm-py3.10 / test (default, 1, 6, linux.rocm.gpu.gfx942.1) (gh) (similar failure)
Process completed with exit code 1.
trunk / linux-jammy-rocm-py3.10 / test (default, 2, 6, linux.rocm.gpu.gfx942.1) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / linux-jammy-rocm-py3.10 / test (default, 3, 6, linux.rocm.gpu.gfx942.1) (gh) (similar failure)
Process completed with exit code 1.
trunk / linux-jammy-rocm-py3.10 / test (default, 4, 6, linux.rocm.gpu.gfx942.1) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / linux-jammy-rocm-py3.10 / test (default, 5, 6, linux.rocm.gpu.gfx942.1) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / linux-jammy-rocm-py3.10 / test (default, 6, 6, linux.rocm.gpu.gfx942.1) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.gfx942.4) (gh) (similar failure)
Process completed with exit code 1.
trunk / linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.gfx942.4) (gh) (similar failure)
Process completed with exit code 1.
trunk / linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4) (gh) (similar failure)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

ghstack-source-id: e327c27 Pull Request resolved: #168013

eellison · 2025-11-18T18:55:49Z

@pytorchbot rebase

pytorchmergebot · 2025-11-18T18:59:07Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-11-18T18:59:14Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict gh/eellison/872/orig returned non-zero exit code 1

Rebasing (1/2)
Auto-merging test/inductor/test_loop_ordering.py
CONFLICT (content): Merge conflict in test/inductor/test_loop_ordering.py
Auto-merging torch/_inductor/codegen/simd.py
Auto-merging torch/_inductor/tiling_utils.py
error: could not apply b9c7ba9a01c... Tiling bug fix
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Could not apply b9c7ba9a01c... # Tiling bug fix

Raised by https://github.com/pytorch/pytorch/actions/runs/19477738026

[ghstack-poisoned]

ghstack-source-id: 5d2ab07 Pull Request resolved: #168013

eellison · 2025-11-19T15:41:25Z

@pytorchbot merge

pytorchmergebot · 2025-11-19T15:48:53Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-11-19T15:50:04Z

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor / inductor-test / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

ghstack-source-id: 5d2ab07 Pull Request resolved: pytorch/pytorch#168013

[ghstack-poisoned]

ghstack-source-id: 4fedfaf Pull Request resolved: #168013

eellison · 2025-12-12T00:57:57Z

@pytorchbot merge

pytorchmergebot · 2025-12-12T00:59:51Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-12-12T01:05:30Z

Merge failed

Reason: 2 jobs have failed, first few of them are: trunk / linux-jammy-rocm-py3.10 / test (default, 3, 6, linux.rocm.gpu.gfx942.1), trunk / linux-jammy-rocm-py3.10 / test (default, 1, 6, linux.rocm.gpu.gfx942.1)

Details for Dev Infra team

Raised by workflow job

eellison · 2025-12-12T01:15:40Z

@pytorchbot merge -i

pytorchmergebot · 2025-12-12T01:17:35Z

Merge started

Your change will be merged while ignoring the following 2 checks: trunk / linux-jammy-rocm-py3.10 / test (default, 3, 6, linux.rocm.gpu.gfx942.1), trunk / linux-jammy-rocm-py3.10 / test (default, 1, 6, linux.rocm.gpu.gfx942.1)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-12-12T07:16:04Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

eellison · 2025-12-12T14:16:46Z

@pytorchbot merge -i "rocm tests failing"

pytorch-bot · 2025-12-12T14:16:49Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: rocm tests failing

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick} ...

Try @pytorchbot --help for more info.

eellison · 2025-12-12T14:35:49Z

@pytorchbot merge -i "all rocm tests failing, unrelated"

pytorch-bot · 2025-12-12T14:35:51Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: all rocm tests failing, unrelated

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick} ...

Try @pytorchbot --help for more info.

eellison · 2025-12-12T14:45:17Z

@pytorchbot merge -i

pytorchmergebot · 2025-12-12T14:47:22Z

Merge started

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ytorch#168013) Another partial fix to pytorch#166653: We had not yet handled multiple reduction vars in tiling splits, which led to the coalesce analysis not seeing the vars as coalesced. See, updated P2043574063, which shows the kernel with correct coalescing by rblock (private link bc private model). Pull Request resolved: pytorch#168013 Approved by: https://github.com/shunting314

Update

e28a625

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: inductor labels Nov 17, 2025

eellison added a commit that referenced this pull request Nov 17, 2025

handle multiple reductions in node splits & read/write normalization

9509c19

ghstack-source-id: b36f98f Pull Request resolved: #168013

eellison requested a review from shunting314 November 17, 2025 17:32

eellison added the topic: not user facing topic category label Nov 17, 2025

Update

c1e5fc4

[ghstack-poisoned]

eellison mentioned this pull request Nov 17, 2025

Tiling bug fix #167771

Closed

eellison added a commit that referenced this pull request Nov 17, 2025

handle multiple reductions in node splits & read/write normalization

46e196a

ghstack-source-id: e327c27 Pull Request resolved: #168013

shunting314 approved these changes Nov 18, 2025

View reviewed changes

Update

a4afa6c

[ghstack-poisoned]

eellison added a commit that referenced this pull request Nov 19, 2025

handle multiple reductions in node splits & read/write normalization

6fc73c2

ghstack-source-id: 5d2ab07 Pull Request resolved: #168013

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 19, 2025

pytorchmergebot added the merging label Nov 19, 2025

pytorchmergebot removed the merging label Nov 19, 2025

eellison mentioned this pull request Nov 19, 2025

Fix cudagraph partitioning handling of deterministic fallbacks #168185

Closed

tiendatngcs pushed a commit to tiendatngcs/pytorch-Dec25 that referenced this pull request Dec 10, 2025

handle multiple reductions in node splits & read/write normalization

79da275

ghstack-source-id: 5d2ab07 Pull Request resolved: pytorch/pytorch#168013

Update

9a2a344

[ghstack-poisoned]

eellison added a commit that referenced this pull request Dec 11, 2025

handle multiple reductions in node splits & read/write normalization

041b25c

ghstack-source-id: 4fedfaf Pull Request resolved: #168013

pytorchmergebot added the merging label Dec 12, 2025

pytorchmergebot removed the merging label Dec 12, 2025

pytorchmergebot added the merging label Dec 12, 2025

pytorchmergebot added the Merged label Dec 12, 2025

pytorchmergebot closed this in 8d2a06b Dec 12, 2025

pytorchmergebot removed the merging label Dec 12, 2025

github-actions bot deleted the gh/eellison/872/head branch January 12, 2026 02:22

Conversation

eellison commented Nov 17, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/168013

✅ You can merge normally! (9 Unrelated Failures)

Uh oh!

eellison commented Nov 18, 2025

Uh oh!

pytorchmergebot commented Nov 18, 2025

Uh oh!

pytorchmergebot commented Nov 18, 2025

Uh oh!

eellison commented Nov 19, 2025

Uh oh!

pytorchmergebot commented Nov 19, 2025

Merge started

Uh oh!

pytorchmergebot commented Nov 19, 2025

Merge failed

Uh oh!

eellison commented Dec 12, 2025

Uh oh!

pytorchmergebot commented Dec 12, 2025

Merge started

Uh oh!

pytorchmergebot commented Dec 12, 2025

Merge failed

Uh oh!

eellison commented Dec 12, 2025

Uh oh!

pytorchmergebot commented Dec 12, 2025

Merge started

Uh oh!

pytorchmergebot commented Dec 12, 2025

Uh oh!

eellison commented Dec 12, 2025

Uh oh!

pytorch-bot bot commented Dec 12, 2025

Uh oh!

eellison commented Dec 12, 2025

Uh oh!

pytorch-bot bot commented Dec 12, 2025

Uh oh!

eellison commented Dec 12, 2025

Uh oh!

pytorchmergebot commented Dec 12, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eellison commented Nov 17, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 17, 2025 •

edited

Loading