Skip to content

[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Backend#176547

Closed
NikhilAPatel wants to merge 9 commits intogh/NikhilAPatel/116/basefrom
gh/NikhilAPatel/116/head
Closed

[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Backend#176547
NikhilAPatel wants to merge 9 commits intogh/NikhilAPatel/116/basefrom
gh/NikhilAPatel/116/head

Conversation

@NikhilAPatel
Copy link
Copy Markdown
Contributor

@NikhilAPatel NikhilAPatel commented Mar 5, 2026 •
edited
Loading

Stack from ghstack (oldest at bottom):

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 5, 2026 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/176547

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 286a853 with merge base 572f0d0 (image):

NEW FAILURE - The following job has failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 5, 2026

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

This was referenced Mar 5, 2026
Closed
Closed
Closed
Closed
Closed
Closed
@NikhilAPatel NikhilAPatel marked this pull request as draft March 5, 2026 01:33
@NikhilAPatel
Update on "[Inductor][NVGEMM] Add blockscaled GEMM wrapper with FP4 s…
4a1e922 
…upport"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel
Update on "[Inductor][NVGEMM] Add blockscaled GEMM wrapper with FP4 s…
0bea065 
…upport"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel NikhilAPatel changed the title [Inductor][NVGEMM] Add blockscaled GEMM wrapper with FP4 support [Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Mar 5, 2026
@NikhilAPatel NikhilAPatel changed the title [Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM [Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Backend Mar 5, 2026
@NikhilAPatel NikhilAPatel marked this pull request as ready for review March 6, 2026 00:04
@NikhilAPatel NikhilAPatel requested a review from mlazos March 6, 2026 00:10
mlazos
mlazos reviewed Mar 8, 2026
View reviewed changes
mlazos
mlazos approved these changes Mar 8, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@mlazos mlazos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One testing comment, otherwise looks good.

@NikhilAPatel
Update on "[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with …
37443c5 
…NVGEMM Backend"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
This was referenced Mar 8, 2026
Closed
Closed
@NikhilAPatel
Update on "[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with …
baa0689 
…NVGEMM Backend"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel NikhilAPatel mentioned this pull request Mar 9, 2026
Closed
@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Mar 9, 2026

Starting merge as part of PR stack under #176549

dshi7 pushed a commit to dshi7/pytorch that referenced this pull request Mar 23, 2026
@NikhilAPatel @dshi7
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
7702b69 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
dshi7 pushed a commit to dshi7/pytorch that referenced this pull request Mar 23, 2026
@NikhilAPatel @dshi7
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
dcd7818 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
dshi7 pushed a commit to dshi7/pytorch that referenced this pull request Mar 23, 2026
@NikhilAPatel @dshi7
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
632c3d9 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
cyyever pushed a commit to cyyever/pytorch that referenced this pull request Mar 24, 2026
@NikhilAPatel @cyyever
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
c8c3728 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Back…
6c1a155 
…end (pytorch#176547)

Pull Request resolved: pytorch#176547
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM (…
d898ccc 
…pytorch#176548)

Pull Request resolved: pytorch#176548
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Restore fp4 kernel docstrings (pytorch#176549)
0a7bfab 
Needed to remove some docstrings from pytorch#176546 in order to fit in the 2000 LoC limit. This PR adds them back.

Pull Request resolved: pytorch#176549
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVG…
7698711 
…EMM Backend (pytorch#176547)"

This reverts commit dd06fff.

Reverted pytorch#176547 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Back…
2238e67 
…end (pytorch#176547)

Pull Request resolved: pytorch#176547
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM (…
a82838f 
…pytorch#176548)

Pull Request resolved: pytorch#176548
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Restore fp4 kernel docstrings (pytorch#176549)
a9debe8 
Needed to remove some docstrings from pytorch#176546 in order to fit in the 2000 LoC limit. This PR adds them back.

Pull Request resolved: pytorch#176549
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
8e8461f 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
9031706 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
8ad652f 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
This was referenced Apr 1, 2026
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
047436c 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
0426dbb 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
eed979a 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
@github-actions github-actions bot deleted the gh/NikhilAPatel/116/head branch April 10, 2026 02:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mlazos mlazos mlazos approved these changes

@albanD albanD Awaiting requested review from albanD

Assignees

No one assigned

Labels

ci-no-td Do not run TD on this PR ciflow/b200 ciflow/inductor ciflow/torchtitan Run TorchTitan integration tests ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor Reverted topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NikhilAPatel @pytorchmergebot @mlazos