[DataTiling] Switch default to start from the DispatchCreation phase. by hanhanW · Pull Request #21441 · iree-org/iree

hanhanW · 2025-07-21T23:08:26Z

The revision updates the default behavior of data-tiling (i.e., --iree-opt-data-tiling flag) to start from the DispatchCreation phase instead of GlobalOptimization phase. The main benefit from the path is that it enables more fusion opportunities and supports multi-devices.

The path that starts from global optimization phase will be remained for a while. Users can get back to the legacy path by using --iree-global-opt-data-tiling flag.

Feel free to try the new path and file issues. We've seen improvements in sdxl clip model on the new path. We're happy to provide guidance if users have any questions about the new standard data-tiling optimization.

Website Doc Preview: https://iree.dev/reference/optimization-options/#global-optimization-iree-global-optimization-opt-level

ci-extra: test_torch

The compiler is very smart on static shape inference that can generate partially dynamic shape during the lowering. It makes data-tiling fusion very struggle because they are expected to be dynamic shape but some dimensions are inferred to static values in Stream AnnotateDispatchAssumptions pass. Because it will lead to `tensor.cast -> set_encoding -> tensor.cast` sequence in a dispatch, while we expect the bindings have encoded tensor types. E.g., Input IR: ```mlir %0 = iree_tensor_ext.dispatch_load ... tensor<?x?xi8> %1 = set_encoding %0 : tensor<?x?xi8> -> tensor<?x?xi8, #encoding> iree_tensor_ext.dispatch_store %1, ... tensor<?x?xi8, #encoding> -> ... tensor<?x?xi8, #encoding> ``` After annotation: ```mlir %0 = iree_tensor_ext.dispatch_load ... tensor<4x?xi8> %cast = tensor.cast %0 : tensor<4x?xi8> -> tensor<?x?xi8> %1 = set_encoding %cast : tensor<?x?xi8> -> tensor<?x?xi8, #encoding> %cast_0 = tensor.cast %1 : tensor<?x?xi8, #encoding> to tensor<4x5xi8> iree_tensor_ext.dispatch_store %cast_0, ... tensor<4x5xi8> -> ... tensor<?x?xi8, #encoding> ``` It is hard to materialize the encodings when cast op is present. Given that the original goal is testing dynamic shape, modifying the input program is an easier fix. The issue is observed from #21441. Signed-off-by: hanhanW <hanhan0912@gmail.com>

hanhanW · 2025-08-11T23:30:34Z

Here is the failing dispatch on GPU side: https://gist.github.com/hanhanW/af6eb4b01ef2be2037e929be223a3fed

I think we need to replace the readonly dest tensor with tensor.empty for the elementwise op.

hanhanW · 2025-10-23T00:14:53Z

Folks, I plan to land the PR by the end of the week. @MaheshRavishankar can you help review the PR?

egebeysel · 2025-10-23T07:28:29Z

I checked how dt fusion looks like with SVE and unfortunately it currently fails, I'll take a deeper dive to the issues later. But as mentioned, that's WIP and obviously not a blocker. I'd be curious to see a performance chart with DT vs DT-fusion though if you have that :)

hanhanW · 2025-10-23T17:26:13Z

I checked how dt fusion looks like with SVE and unfortunately it currently fails, I'll take a deeper dive to the issues later. But as mentioned, that's WIP and obviously not a blocker. I'd be curious to see a performance chart with DT vs DT-fusion though if you have that :)

Does it work with the original path? I thought that there are vscale ops, so it is not working.

hanhanW · 2025-10-23T17:28:32Z

The only perf data I have is from CI: https://github.com/iree-org/iree/actions/runs/18733272191?pr=21441

Without the patch:

With the patch:

The binary size drops a lot after rebase, I'll update it.

JerryShih · 2025-10-27T02:00:19Z

@hanhanW
Will the const-eval still work when the tiling is moving to dispatch-creation pass(especially for mmt4d RHS/LHS packing)?

Max191 · 2025-10-27T13:45:31Z

@hanhanW Will the const-eval still work when the tiling is moving to dispatch-creation pass(especially for mmt4d RHS/LHS packing)?

Data tiling starting from dispatch creation will use the late materialization path, and const-eval is not yet implemented for late materialization, but const-expr-hoisting should still work.

hanhanW · 2025-10-27T19:35:17Z

I may start the const-eval work after I finish the memory footprint issue. They are not directly related, but they are all important for data-tiling. Otherwise, it may crash when data-tiling is enabled.

Here is the initial breakdown for const-eval work: https://discord.com/channels/689900678990135345/1428131175528136764 The intention was having a broader discussion in the thread, but it is not happening. I will open an issue for it when I get cycles.

hanhanW · 2025-11-05T23:16:31Z

rebase to trigger new torch test suites, I'll update the golden values in a follow-up commit (within the PR).

hanhanW · 2025-11-07T22:30:35Z

We recently improved CPU backends. The golden value for the clip is 141 ms (old path) v.s. 133 ms (new path), FYI.

hanhanW · 2025-11-08T05:52:33Z

I'll land this on next Wed.

Signed-off-by: hanhanW <hanhan0912@gmail.com>

hanhanW · 2025-11-13T04:12:27Z

I can't land this because shark10 does not pick up the job today... I think it is dead, and I already reached out to POC who can help with the machine issue.

Signed-off-by: hanhanW <hanhan0912@gmail.com>

…n phase. (#21441)" This reverts commit 392931d.

…iree-org#21441) The revision updates the default behavior of data-tiling (i.e., `--iree-opt-data-tiling` flag) to start from the **DispatchCreation** phase instead of **GlobalOptimization** phase. The main benefit from the path is that it enables more fusion opportunities and supports multi-devices. The path that starts from global optimization phase will be remained for a while. Users can get back to the legacy path by using `--iree-global-opt-data-tiling` flag. Feel free to try the new path and file issues. We've seen improvements in sdxl clip model on the new path. We're happy to provide guidance if users have any questions about the new standard data-tiling optimization. Website Doc Preview: https://iree.dev/reference/optimization-options/#global-optimization-iree-global-optimization-opt-level ci-extra: test_torch --------- Signed-off-by: hanhanW <hanhan0912@gmail.com>

The compiler is very smart on static shape inference that can generate partially dynamic shape during the lowering. It makes data-tiling fusion very struggle because they are expected to be dynamic shape but some dimensions are inferred to static values in Stream AnnotateDispatchAssumptions pass. Because it will lead to `tensor.cast -> set_encoding -> tensor.cast` sequence in a dispatch, while we expect the bindings have encoded tensor types. E.g., Input IR: ```mlir %0 = iree_tensor_ext.dispatch_load ... tensor<?x?xi8> %1 = set_encoding %0 : tensor<?x?xi8> -> tensor<?x?xi8, #encoding> iree_tensor_ext.dispatch_store %1, ... tensor<?x?xi8, #encoding> -> ... tensor<?x?xi8, #encoding> ``` After annotation: ```mlir %0 = iree_tensor_ext.dispatch_load ... tensor<4x?xi8> %cast = tensor.cast %0 : tensor<4x?xi8> -> tensor<?x?xi8> %1 = set_encoding %cast : tensor<?x?xi8> -> tensor<?x?xi8, #encoding> %cast_0 = tensor.cast %1 : tensor<?x?xi8, #encoding> to tensor<4x5xi8> iree_tensor_ext.dispatch_store %cast_0, ... tensor<4x5xi8> -> ... tensor<?x?xi8, #encoding> ``` It is hard to materialize the encodings when cast op is present. Given that the original goal is testing dynamic shape, modifying the input program is an easier fix. The issue is observed from #21441. Signed-off-by: hanhanW <hanhan0912@gmail.com>

This was referenced Jul 21, 2025

[DT] Disable early materialization by default. #20323

Closed

[NFC] Switch dynamic inputs to flow.tensor.dynamic_constant. #21461

Merged

hanhanW force-pushed the dt-fusion-default branch 4 times, most recently from efb04d1 to 8932f5e Compare July 29, 2025 22:13

hanhanW force-pushed the dt-fusion-default branch 6 times, most recently from c8f64d5 to 510440c Compare August 6, 2025 18:13

hanhanW force-pushed the dt-fusion-default branch 5 times, most recently from 370c5d9 to 9b48bf9 Compare August 11, 2025 22:44

hanhanW force-pushed the dt-fusion-default branch 2 times, most recently from 9ce9a06 to 7c370dd Compare August 12, 2025 23:55

hanhanW mentioned this pull request Aug 13, 2025

[DT][CPU] Exclude pack ops with reshape producers from lowering config setting #21675

Merged

hanhanW force-pushed the dt-fusion-default branch 5 times, most recently from 946c2d8 to 02a3f01 Compare August 14, 2025 21:09

hanhanW force-pushed the dt-fusion-default branch 2 times, most recently from b15c16f to 7f03dc3 Compare August 22, 2025 17:28

hanhanW force-pushed the dt-fusion-default branch from 7f03dc3 to 6fb349d Compare September 2, 2025 21:12

hanhanW mentioned this pull request Oct 20, 2025

Release Tracker v3.9.0 - (2025-11-24) #22307

Open

7 tasks

hanhanW requested a review from MaheshRavishankar October 23, 2025 00:14

hanhanW force-pushed the dt-fusion-default branch from 93a0e98 to b252bb1 Compare November 7, 2025 22:28

MaheshRavishankar approved these changes Nov 7, 2025

View reviewed changes

hanhanW force-pushed the dt-fusion-default branch from b252bb1 to 2b40d89 Compare November 12, 2025 19:52

hanhanW added 4 commits November 12, 2025 14:10

Adjust benchmark golden time/dispatch/size

6a9586b

Signed-off-by: hanhanW <hanhan0912@gmail.com>

[DataTiling] Switch default to start from the DispatchCreation phase.

37f0559

Signed-off-by: hanhanW <hanhan0912@gmail.com>

Refresh golden values because of local encoding propagation.

ad025cf

Signed-off-by: hanhanW <hanhan0912@gmail.com>

Update docs

17f36fb

Signed-off-by: hanhanW <hanhan0912@gmail.com>

hanhanW force-pushed the dt-fusion-default branch from 2b40d89 to 17f36fb Compare November 12, 2025 22:11

hanhanW added 2 commits November 13, 2025 15:10

refresh golden numbers

9d7cefa

Signed-off-by: hanhanW <hanhan0912@gmail.com>

Merge branch 'main' into dt-fusion-default

5719800

hanhanW merged commit 392931d into iree-org:main Nov 14, 2025
50 of 52 checks passed

hanhanW deleted the dt-fusion-default branch November 14, 2025 18:32

lialan added a commit that referenced this pull request Nov 17, 2025

Revert "[DataTiling] Switch default to start from the DispatchCreatio…

45585ee

…n phase. (#21441)" This reverts commit 392931d.

hanhanW mentioned this pull request Nov 19, 2025

RFC: Switching data-tiling to late materialization path #19967

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DataTiling] Switch default to start from the DispatchCreation phase.#21441

[DataTiling] Switch default to start from the DispatchCreation phase.#21441
hanhanW merged 6 commits intoiree-org:mainfrom
hanhanW:dt-fusion-default

hanhanW commented Jul 21, 2025 •

edited

Loading

Uh oh!

hanhanW commented Aug 11, 2025

Uh oh!

hanhanW commented Oct 23, 2025

Uh oh!

egebeysel commented Oct 23, 2025

Uh oh!

hanhanW commented Oct 23, 2025

Uh oh!

hanhanW commented Oct 23, 2025

Uh oh!

JerryShih commented Oct 27, 2025

Uh oh!

Max191 commented Oct 27, 2025

Uh oh!

hanhanW commented Oct 27, 2025

Uh oh!

hanhanW commented Nov 5, 2025

Uh oh!

hanhanW commented Nov 7, 2025

Uh oh!

hanhanW commented Nov 8, 2025

Uh oh!

hanhanW commented Nov 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

hanhanW commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanhanW commented Aug 11, 2025

Uh oh!

hanhanW commented Oct 23, 2025

Uh oh!

egebeysel commented Oct 23, 2025

Uh oh!

hanhanW commented Oct 23, 2025

Uh oh!

hanhanW commented Oct 23, 2025

Uh oh!

JerryShih commented Oct 27, 2025

Uh oh!

Max191 commented Oct 27, 2025

Uh oh!

hanhanW commented Oct 27, 2025

Uh oh!

hanhanW commented Nov 5, 2025

Uh oh!

hanhanW commented Nov 7, 2025

Uh oh!

hanhanW commented Nov 8, 2025

Uh oh!

hanhanW commented Nov 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hanhanW commented Jul 21, 2025 •

edited

Loading