Skip to content

[inductor] support linear & layer_norm unbacked#155267

Closed
ColinPeppler wants to merge 5 commits intogh/ColinPeppler/72/basefrom
gh/ColinPeppler/72/head
Closed

[inductor] support linear & layer_norm unbacked#155267
ColinPeppler wants to merge 5 commits intogh/ColinPeppler/72/basefrom
gh/ColinPeppler/72/head

Conversation

@ColinPeppler
Copy link
Contributor

@ColinPeppler ColinPeppler commented Jun 5, 2025

What

  • Use statically_known_true over guard_size_oblivious in cases where we're checking an optimization path. Otherwise, it will DDE and we can't take the safe/slower path.
  • For broadcast checks, use fallback=False if we encounter a DDE. Typically, unbackeds would be ≥2 and that falls inline with size-oblivious reasoning (i.e. when size_oblivious=True).

Example DDE

torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/lowering.py:488 in broadcast_symbolic_shapes)
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/ir.py:2797 in create)

Stack from ghstack (oldest at bottom):

cc @ezyang @penguinwu @bobrenjc93 @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

@pytorch-bot
Copy link

pytorch-bot bot commented Jun 5, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155267

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ You can merge normally! (1 Unrelated Failure)

As of commit b37ff27 with merge base 25fbf09 (image):

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ColinPeppler added a commit that referenced this pull request Jun 5, 2025
ghstack-source-id: 628173e
Pull Request resolved: #155267
@ColinPeppler ColinPeppler added the topic: not user facing topic category label Jun 5, 2025
Copy link
Contributor

@eellison eellison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @zou3519 , @bdhirsh - what's our policy for this ? How do we reconcile "strides wrong is a ubn" and now introducing meta functions where we are potentially not computing strides correctly.

@zou3519
Copy link
Contributor

zou3519 commented Jun 17, 2025

cc @zou3519 , @bdhirsh - what's our policy for this ? How do we reconcile "strides wrong is a ubn" and now introducing meta functions where we are potentially not computing strides correctly.

We discussed this at composability sync. The conclusion was:

  1. strides wrong for backed symints is a UBN
  2. unbacked symints are explicitly wrong. There's no way to fulfill the requirements for unbacked symints and have correct strides at the same time. So unbacked symints being "wrong" is not a problem.

### What
- Use `statically_known_true` over `guard_size_oblivious` in cases where we're checking an optimization path. Otherwise, it will DDE and we can't take the safe/slower path.
- For broadcast checks, use fallback=False if we encounter a DDE. Typically, unbackeds would be ≥2 and that falls inline with size-oblivious reasoning.
- In `coerce_tangent_and_suggest_memory_format`, use the torch._refs version of `contiguous` over Aten's version because Aten doesn't have a DDE-friendly path (yet).

### Example DDE
```
GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(128*((u0//387)), 0) (unhinted: Eq(128*((u0//387)), 0)).  (Size-like symbols: u0)

Caused by: (_functorch/_aot_autograd/collect_metadata_analysis.py:84 in coerce_tangent_and_suggest_memory_format)
```

```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/lowering.py:488 in broadcast_symbolic_shapes)
```
```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/ir.py:2797 in create)
```




cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov

[ghstack-poisoned]
ColinPeppler added a commit that referenced this pull request Jul 18, 2025
ghstack-source-id: bdccfdc
Pull Request resolved: #155267
### What
- Use `statically_known_true` over `guard_size_oblivious` in cases where we're checking an optimization path. Otherwise, it will DDE and we can't take the safe/slower path.
- For broadcast checks, use fallback=False if we encounter a DDE. Typically, unbackeds would be ≥2 and that falls inline with size-oblivious reasoning.
- In `coerce_tangent_and_suggest_memory_format`, use the torch._refs version of `contiguous` over Aten's version because Aten doesn't have a DDE-friendly path (yet).

### Example DDE
```
GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(128*((u0//387)), 0) (unhinted: Eq(128*((u0//387)), 0)).  (Size-like symbols: u0)

Caused by: (_functorch/_aot_autograd/collect_metadata_analysis.py:84 in coerce_tangent_and_suggest_memory_format)
```

```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/lowering.py:488 in broadcast_symbolic_shapes)
```
```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/ir.py:2797 in create)
```




cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov

[ghstack-poisoned]
ColinPeppler added a commit that referenced this pull request Jul 18, 2025
ghstack-source-id: 902c7d0
Pull Request resolved: #155267
@ColinPeppler ColinPeppler requested a review from laithsakka July 21, 2025 19:51
@laithsakka
Copy link
Contributor

@eellison the old used to used size_oblivious=True anyway so this is not changing existing behaviour for unbacked anwyay size_oblivious=True means assume unbacked>=2

(
V.graph.sizevars.shape_env.evaluate_expr(
sympy.Eq(a, 1), size_oblivious=True
sympy.Eq(a, 1), size_oblivious=True, fallback_value=False
Copy link
Contributor

@laithsakka laithsakka Jul 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove size_oblivious=True ditto for others.

### What
- Use `statically_known_true` over `guard_size_oblivious` in cases where we're checking an optimization path. Otherwise, it will DDE and we can't take the safe/slower path.
- For broadcast checks, use fallback=False if we encounter a DDE. Typically, unbackeds would be ≥2 and that falls inline with size-oblivious reasoning.

### Example DDE
```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/lowering.py:488 in broadcast_symbolic_shapes)
```
```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/ir.py:2797 in create)
```




cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov

[ghstack-poisoned]
ColinPeppler added a commit that referenced this pull request Jul 22, 2025
ghstack-source-id: db1a366
Pull Request resolved: #155267
@eellison eellison requested review from bobrenjc93 and ezyang July 22, 2025 14:59
Copy link
Contributor

@eellison eellison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable

Comment on lines 945 to 958
V.graph.sizevars.shape_env.evaluate_expr(
sympy.Eq(a, 1), size_oblivious=True
sympy.Eq(a, 1), fallback_value=False
)
and not V.graph.sizevars.shape_env.evaluate_expr(
sympy.Eq(b, 1), size_oblivious=True
sympy.Eq(b, 1), fallback_value=False
)
)
or (
not V.graph.sizevars.shape_env.evaluate_expr(
sympy.Eq(a, 1), size_oblivious=True
sympy.Eq(a, 1), fallback_value=False
)
and V.graph.sizevars.shape_env.evaluate_expr(
sympy.Eq(b, 1), size_oblivious=True
sympy.Eq(b, 1), fallback_value=False
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reuse the above broadcast_symbolic_shapes here instead of duplicating the logic ? (i know this was preexisting to your change)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one would look a lil bit forced since broadcast_symbolic_shapes returns the broadcasted shape vs. here we just want to check for broadcast.

But agreed the code looks duplicated, I'm gonna find a more reusable way.

### What
- Use `statically_known_true` over `guard_size_oblivious` in cases where we're checking an optimization path. Otherwise, it will DDE and we can't take the safe/slower path.
- For broadcast checks, use `fallback=False` if we encounter a DDE. Typically, unbackeds would be ≥2 and that falls inline with size-oblivious reasoning (i.e. when `size_oblivious=True`).

### Example DDE
```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/lowering.py:488 in broadcast_symbolic_shapes)
```
```
torch._inductor.exc.InductorError: LoweringException: GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq((u0//387), 1) (unhinted: Eq((u0//387), 1)).  (Size-like symbols: u0)

Caused by: (_inductor/ir.py:2797 in create)
```




cc ezyang penguinwu bobrenjc93 voznesenskym EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben

[ghstack-poisoned]
ColinPeppler added a commit that referenced this pull request Jul 22, 2025
ghstack-source-id: 2c714f5
Pull Request resolved: #155267
@ColinPeppler
Copy link
Contributor Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 23, 2025
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/ColinPeppler/72/head branch August 23, 2025 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants