Skip to content

[Inductor][NVGEMM] Add infrastructure for registering custom kernels with NVGEMM Cutlass API#176543

Closed
NikhilAPatel wants to merge 3 commits intogh/NikhilAPatel/112/basefrom
gh/NikhilAPatel/112/head
Closed

[Inductor][NVGEMM] Add infrastructure for registering custom kernels with NVGEMM Cutlass API#176543
NikhilAPatel wants to merge 3 commits intogh/NikhilAPatel/112/basefrom
gh/NikhilAPatel/112/head

Conversation

@NikhilAPatel
Copy link
Copy Markdown
Contributor

@NikhilAPatel NikhilAPatel commented Mar 5, 2026 •
edited
Loading

Stack from ghstack (oldest at bottom):

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo

@NikhilAPatel
[Inductor][NVGEMM] Add infrastructure for vendored CuTeDSL kernel wra…
ef74a30 
…ppers

Authored with Claude.

[ghstack-poisoned]
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 5, 2026 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/176543

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Cancelled Job, 3 Unrelated Failures

As of commit ff29833 with merge base 572f0d0 (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOB - The following job was cancelled. Please retry:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 5, 2026

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

This was referenced Mar 5, 2026
Closed
Closed
Closed
Closed
Closed
Closed
@NikhilAPatel NikhilAPatel marked this pull request as draft March 5, 2026 01:35
@NikhilAPatel
Update on "[Inductor][NVGEMM] Add infrastructure for vendored CuTeDSL…
76efd60 
… kernel wrappers"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel NikhilAPatel changed the title [Inductor][NVGEMM] Add infrastructure for vendored CuTeDSL kernel wrappers [Inductor][NVGEMM] Add infrastructure for registering custom kernels with NVGEMM Cutlass API Mar 5, 2026
@NikhilAPatel NikhilAPatel marked this pull request as ready for review March 6, 2026 00:03
@NikhilAPatel NikhilAPatel requested a review from mlazos March 6, 2026 00:03
This was referenced Mar 8, 2026
Closed
Closed
Closed
@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Mar 9, 2026

Starting merge as part of PR stack under #176549

pytorchmergebot pushed a commit that referenced this pull request Mar 9, 2026
@NikhilAPatel @pytorchmergebot
[Inductor] Add FP4 tensor creation support for autotuning (#176544)
eb97b58 
Pull Request resolved: #176544
Approved by: https://github.com/mlazos
ghstack dependencies: #176543
pytorchmergebot pushed a commit that referenced this pull request Mar 9, 2026
@NikhilAPatel @pytorchmergebot
[Inductor][NVGEMM] Patch cutlass_api FP4 dtype mapping (#176545)
8df497d 
Pull Request resolved: #176545
Approved by: https://github.com/mlazos
ghstack dependencies: #176543, #176544
pytorchmergebot pushed a commit that referenced this pull request Mar 9, 2026
@NikhilAPatel @pytorchmergebot
[Inductor][NVGEMM] Add CuTeDSL Blockscaled GEMM Kernel (#176546)
c4fd6ef 
Instead of cloning this directly from the Cutlass repo via `setup.py`, we need to own it ourselves inside of Inductor to do some Tensor mode reordering due to the differences between how Inductor and this kernel need the dims ordered

Pull Request resolved: #176546
Approved by: https://github.com/mlazos
ghstack dependencies: #176543, #176544, #176545
pytorchmergebot pushed a commit that referenced this pull request Mar 9, 2026
@NikhilAPatel @pytorchmergebot
[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Back…
dd06fff 
…end (#176547)

Pull Request resolved: #176547
Approved by: https://github.com/mlazos
ghstack dependencies: #176543, #176544, #176545, #176546
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blocksca…
63e3859 
…led GEMM (pytorch#176548)"

This reverts commit 9db6179.

Reverted pytorch#176548 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVG…
7698711 
…EMM Backend (pytorch#176547)"

This reverts commit dd06fff.

Reverted pytorch#176547 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Add CuTeDSL Blockscaled GEMM Kernel (pytor…
1feb862 
…ch#176546)"

This reverts commit c4fd6ef.

Reverted pytorch#176546 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Patch cutlass_api FP4 dtype mapping (pytor…
a49bc15 
…ch#176545)"

This reverts commit 8df497d.

Reverted pytorch#176545 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor] Add FP4 tensor creation support for autotuning (py…
f6dbd5c 
…torch#176544)"

This reverts commit eb97b58.

Reverted pytorch#176544 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Add infrastructure for registering custom …
d2afd7f 
…kernels with NVGEMM Cutlass API (pytorch#176543)"

This reverts commit 9e49f44.

Reverted pytorch#176543 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Add infrastructure for registering custom kernels …
64902e7 
…with NVGEMM Cutlass API (pytorch#176543)

Pull Request resolved: pytorch#176543
Approved by: https://github.com/mlazos
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor] Add FP4 tensor creation support for autotuning (pytorch#17…
6a291c8 
…6544)

Pull Request resolved: pytorch#176544
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Patch cutlass_api FP4 dtype mapping (pytorch#176545)
b85aeac 
Pull Request resolved: pytorch#176545
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Add CuTeDSL Blockscaled GEMM Kernel (pytorch#176546)
5e3dd59 
Instead of cloning this directly from the Cutlass repo via `setup.py`, we need to own it ourselves inside of Inductor to do some Tensor mode reordering due to the differences between how Inductor and this kernel need the dims ordered

Pull Request resolved: pytorch#176546
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Register CuTeDSL Blockscaled GEMM with NVGEMM Back…
2238e67 
…end (pytorch#176547)

Pull Request resolved: pytorch#176547
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM (…
a82838f 
…pytorch#176548)

Pull Request resolved: pytorch#176548
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Restore fp4 kernel docstrings (pytorch#176549)
a9debe8 
Needed to remove some docstrings from pytorch#176546 in order to fit in the 2000 LoC limit. This PR adds them back.

Pull Request resolved: pytorch#176549
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
8e8461f 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
9031706 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
8ad652f 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
This was referenced Apr 1, 2026
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
@benediktjohannes benediktjohannes mentioned this pull request Apr 3, 2026
Closed
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
047436c 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mlazos mlazos mlazos approved these changes

@albanD albanD Awaiting requested review from albanD

Assignees

No one assigned

Labels

ci-no-td Do not run TD on this PR ciflow/b200 ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor Reverted topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@NikhilAPatel @pytorchmergebot @zou3519 @mlazos