Skip to content

[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM#176548

Closed
NikhilAPatel wants to merge 9 commits intogh/NikhilAPatel/117/basefrom
gh/NikhilAPatel/117/head
Closed

[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM#176548
NikhilAPatel wants to merge 9 commits intogh/NikhilAPatel/117/basefrom
gh/NikhilAPatel/117/head

Conversation

@NikhilAPatel
Copy link
Copy Markdown
Contributor

@NikhilAPatel NikhilAPatel commented Mar 5, 2026 •
edited
Loading

Stack from ghstack (oldest at bottom):

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 5, 2026 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/176548

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit af585a9 with merge base 572f0d0 (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 5, 2026

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

This was referenced Mar 5, 2026
Closed
Closed
Closed
Closed
Closed
Closed
@NikhilAPatel NikhilAPatel marked this pull request as draft March 5, 2026 01:33
@NikhilAPatel
Update on "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 block…
0fd36b7 
…scaled GEMM"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel
Update on "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 block…
07cd792 
…scaled GEMM"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel NikhilAPatel marked this pull request as ready for review March 6, 2026 00:04
@NikhilAPatel NikhilAPatel requested a review from mlazos March 6, 2026 00:10
mlazos
mlazos approved these changes Mar 8, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@mlazos mlazos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we test this as well?

@NikhilAPatel
Update on "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 block…
5d034c3 
…scaled GEMM"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
This was referenced Mar 8, 2026
Closed
Closed
@NikhilAPatel
Update on "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 block…
91f07e7 
…scaled GEMM"

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy kadeng muchulee8 amjames chauhang aakhundov coconutruben jataylo

[ghstack-poisoned]
@NikhilAPatel NikhilAPatel mentioned this pull request Mar 9, 2026
Closed
@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Mar 9, 2026

Starting merge as part of PR stack under #176549

pytorchmergebot pushed a commit that referenced this pull request Mar 9, 2026
@NikhilAPatel @pytorchmergebot
[Inductor][NVGEMM] Restore fp4 kernel docstrings (#176549)
c84f1cb 
Needed to remove some docstrings from #176546 in order to fit in the 2000 LoC limit. This PR adds them back.

Pull Request resolved: #176549
Approved by: https://github.com/mlazos
ghstack dependencies: #176543, #176544, #176545, #176546, #176547, #176548
pytorchmergebot added a commit that referenced this pull request Mar 9, 2026
@pytorchmergebot
Revert "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blocksca…
acf2f1c 
…led GEMM (#176548)"

This reverts commit 9db6179.

Reverted #176548 on behalf of https://github.com/zou3519 due to broke CI ([comment](#176543 (comment)))
pytorchmergebot pushed a commit that referenced this pull request Mar 22, 2026
@NikhilAPatel @pytorchmergebot
[Inductor][NVGEMM] Refactor tests (#176847)
e50dd94 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: #176847
Approved by: https://github.com/mlazos
ghstack dependencies: #176543, #176544, #176545, #176546, #176547, #176548, #176549, #176845
pytorchmergebot pushed a commit that referenced this pull request Mar 22, 2026
@NikhilAPatel @pytorchmergebot
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (#…
3f14378 
…176859)

Pull Request resolved: #176859
Approved by: https://github.com/mlazos
ghstack dependencies: #176543, #176544, #176545, #176546, #176547, #176548, #176549, #176845, #176847
dshi7 pushed a commit to dshi7/pytorch that referenced this pull request Mar 23, 2026
@NikhilAPatel @dshi7
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
7702b69 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
dshi7 pushed a commit to dshi7/pytorch that referenced this pull request Mar 23, 2026
@NikhilAPatel @dshi7
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
dcd7818 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
dshi7 pushed a commit to dshi7/pytorch that referenced this pull request Mar 23, 2026
@NikhilAPatel @dshi7
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
632c3d9 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
cyyever pushed a commit to cyyever/pytorch that referenced this pull request Mar 24, 2026
@NikhilAPatel @cyyever
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
c8c3728 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM (…
d898ccc 
…pytorch#176548)

Pull Request resolved: pytorch#176548
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Restore fp4 kernel docstrings (pytorch#176549)
0a7bfab 
Needed to remove some docstrings from pytorch#176546 in order to fit in the 2000 LoC limit. This PR adds them back.

Pull Request resolved: pytorch#176549
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@pytorchmergebot @EmanueleCoradin
Revert "[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blocksca…
63e3859 
…led GEMM (pytorch#176548)"

This reverts commit 9db6179.

Reverted pytorch#176548 on behalf of https://github.com/zou3519 due to broke CI ([comment](pytorch#176543 (comment)))
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Enable nvMatmulHeuristics for FP4 blockscaled GEMM (…
a82838f 
…pytorch#176548)

Pull Request resolved: pytorch#176548
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547
EmanueleCoradin pushed a commit to EmanueleCoradin/pytorch that referenced this pull request Mar 30, 2026
@NikhilAPatel @EmanueleCoradin
[Inductor][NVGEMM] Restore fp4 kernel docstrings (pytorch#176549)
a9debe8 
Needed to remove some docstrings from pytorch#176546 in order to fit in the 2000 LoC limit. This PR adds them back.

Pull Request resolved: pytorch#176549
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
8e8461f 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
9031706 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
AaronWang04 pushed a commit to AaronWang04/pytorch that referenced this pull request Mar 31, 2026
@NikhilAPatel @AaronWang04
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
8ad652f 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
This was referenced Apr 1, 2026
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
Draft
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor][NVGEMM] Drop tile_k from nvMatmulHeuristics matching (pyto…
047436c 
…rch#176845)

Pull Request resolved: pytorch#176845
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor][NVGEMM] Refactor tests (pytorch#176847)
0426dbb 
  Shared helpers (module-level):
  - _round_up — deduplicated from two test methods
  - _prep_k — deduplicated from two test methods
  - _create_tensor_with_layout — unified layout creation for all dtypes (float16, bf16, fp8, fp4)
  - _nvgemm_config — standard config dict with autotune_fallback_to_aten: False always set

  Bug fixes:
  - Added missing torch._dynamo.reset() to test_scaled_gemm_mxfp8, test_scaled_gemm_nvf4, test_grouped_gemm, test_grouped_gemm_varying_offsets
  - ceildiv moved to top-level import (was imported inside two test methods)

  New test coverage:
  - test_matmul: added ("contiguous", "aligned_offset") and ("contiguous", "padded") layout combos (7 combos, up from 5)
  - test_scaled_gemm_mxfp8: added shape parametrization (4 shapes, was 1)
  - test_grouped_gemm: added layout_a parametrization (contiguous, aligned_offset, view, padded)
  - test_grouped_gemm_varying_offsets: split out from original test_grouped_gemm — tests different offset distributions separately
  - test_fp8_heuristic_configs: new heuristics integration test for FP8 precision strings

  Consistency fixes:
  - All tests now use _nvgemm_config() with autotune_fallback_to_aten: False
  - TestNVUniversalGemmDynamicShapes also uses _nvgemm_config()

Pull Request resolved: pytorch#176847
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845
nklshy-aws pushed a commit to nklshy-aws/pytorch that referenced this pull request Apr 7, 2026
@NikhilAPatel @nklshy-aws
[Inductor] Fix benchmark_example_value losing dtype on view unwrap (p…
eed979a 
…ytorch#176859)

Pull Request resolved: pytorch#176859
Approved by: https://github.com/mlazos
ghstack dependencies: pytorch#176543, pytorch#176544, pytorch#176545, pytorch#176546, pytorch#176547, pytorch#176548, pytorch#176549, pytorch#176845, pytorch#176847
@github-actions github-actions bot deleted the gh/NikhilAPatel/117/head branch April 10, 2026 02:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mlazos mlazos mlazos approved these changes

Assignees

No one assigned

Labels

ci-no-td Do not run TD on this PR ciflow/b200 ciflow/inductor ciflow/torchtitan Run TorchTitan integration tests ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor Reverted topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NikhilAPatel @pytorchmergebot @mlazos