[PERF] Allreduce fusion. Support torch native matching. Tuning of the thresholds by ilmarkov · Pull Request #24248 · vllm-project/vllm

ilmarkov · 2025-09-04T12:23:45Z

First part of improvement on fused allreduce.

Purpose

Add tunings of thresholds for Flashinfer allreduce fusion.

Adds a benchmark for allreduce fusion to determine input size thresholds for flashinfer allreduce.
Updates thresholds for flashinfer allreduce (as well as adding two shot algorithm usage when it has better performance) on Hopper and Blackwell devices

Moves allreduce out of moe_forward custom op in order to be able to match for fusion for moe models.

Test Plan

Added tests for non custom ops fusion
Added e2e test for Qwen3 MoE

Based on #24604
Second part: #24252 Introduce compile ranges

mergify · 2025-09-04T12:24:52Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ilmarkov.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

gemini-code-assist

Code Review

This pull request is a significant enhancement to the all-reduce fusion capabilities, adding support for matching native PyTorch operations in addition to custom ops. This greatly improves usability and performance flexibility. The introduction of a comprehensive benchmark for tuning fusion thresholds is also a valuable addition. The changes are extensive, particularly with the large number of new fusion patterns in vllm/compilation/collective_fusion.py. While the overall approach is sound, I've identified several critical issues in the implementation of these new patterns. Specifically, the return values from some pattern and replacement functions appear to be incorrect, which could lead to fusion failures or incorrect model outputs. I've provided detailed comments and suggestions for these issues. The configuration updates and the new benchmark script are well-implemented and welcome improvements.

gemini-code-assist · 2025-09-04T12:27:26Z

vllm/compilation/collective_fusion.py

The return values from the replacement function are incorrect. The pattern returns (rms_output, allreduce_output), which correspond to the normalized output and the all-reduced tensor. The replacement function should return the same structure.

auto_functionalized(flashinfer_trtllm_fused_allreduce_norm, ...) returns a tuple of 5 mutated arguments: (allreduce_in, residual, norm_out, quant_out, scale_out).

The rms_result corresponds to norm_out, which is allreduce[2].
The allreduce_in (which is input to the replacement function) corresponds to allreduce[0].

Therefore, the return statement should be return allreduce[2], allreduce[0].

The current code returns allreduce[3], allreduce[1], which corresponds to (quant_out, residual). This is incorrect and will lead to fusion failures or wrong results.

Suggested change

return allreduce[3], allreduce[1]

return allreduce[2], allreduce[0]

gemini-code-assist · 2025-09-04T12:27:26Z

vllm/compilation/collective_fusion.py

The return values from the replacement function are incorrect. The pattern returns (rms_output, rms_residual), which are the normalized output and the residual output. The replacement function should return the same structure.

When norm_out=None is passed to flashinfer_trtllm_fused_allreduce_norm, the allreduce_in tensor is used as the output buffer for the normalization result and is mutated. auto_functionalized will return a tuple where the first element (allreduce[0]) is the mutated allreduce_in (i.e., norm_out), and the second element (allreduce[1]) is the mutated residual.

Therefore, the correct return should be return allreduce[0], allreduce[1].

The current code returns allreduce[1], allreduce[2], which corresponds to (residual, norm_out). Since norm_out is None in the call, this is incorrect.

Suggested change

return allreduce[1], allreduce[2]

return allreduce[0], allreduce[1]

nvpohanh · 2025-09-05T01:18:28Z

vllm/compilation/collective_fusion.py

Just curious: why is the threshold still so low for TP8? I think AR+Norm should have pretty good perf up to some larger message sizes for TP8?

vllm/compilation/collective_fusion.py

nvpohanh · 2025-09-05T01:24:01Z

vllm/config/compilation.py

why is it 1MB for TP8?

@nvpohanh Here are the results for TP=8 Blackwell with torch symm mem (VLLM_ALLREDUCE_USE_SYMM_MEM=1) enabled (see the set of results below). I used the best performant alternative to fused allreduce. Probably, we can condition on it checking if symm mem is available and enabled, it will overcomplicate the configuration, in my opinion. Compared default allreduce flashinfer fused alternative is not significantly better in 4-16MB region (see results in the end)

Symm mem enabled

World Size: 8
Hidden Dimension: 8192
Warmup Iterations: 5
Benchmark Trials: 20
Quantization Mode: none

Configuration: seq_len=32, dtype=bfloat16, no residual

Input Size: 0.50 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.029 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.030 0.99x

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.012 2.39x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.086 0.34x

Configuration: seq_len=64, dtype=bfloat16, no residual

Input Size: 1.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.030 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.030 0.99x

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.018 1.62x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.056 0.54x

Configuration: seq_len=128, dtype=bfloat16, no residual

Input Size: 2.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.023 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.024 0.99x

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.033 0.71x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.052 0.45x

Configuration: seq_len=256, dtype=bfloat16, no residual

Input Size: 4.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.031 0.97x

Standard Allreduce Rmsnorm Native Compiled 0.030 baseline

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.064 0.47x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.050 0.60x

Configuration: seq_len=256, dtype=bfloat16, no residual

Input Size: 4.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.031 0.97x

Standard Allreduce Rmsnorm Native Compiled 0.030 baseline

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.049 0.61x

Configuration: seq_len=512, dtype=bfloat16, no residual

Input Size: 8.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.044 0.98x

Standard Allreduce Rmsnorm Native Compiled 0.043 baseline

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.297 0.15x

Configuration: seq_len=1024, dtype=bfloat16, no residual

Input Size: 16.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.071 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.077 0.93x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.109 0.66x

Configuration: seq_len=2048, dtype=bfloat16, no residual

Input Size: 32.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.135 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.143 0.94x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.205 0.66x

Default allreduce

Configuration: seq_len=32, dtype=bfloat16, no residual

Input Size: 0.50 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.029 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.030 0.99x

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.012 2.44x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.087 0.34x

Configuration: seq_len=64, dtype=bfloat16, no residual

Input Size: 1.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.030 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.030 1.00x

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.019 1.63x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.056 0.54x

Configuration: seq_len=128, dtype=bfloat16, no residual

Input Size: 2.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.032 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.032 1.00x

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.033 0.97x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.052 0.62x

Configuration: seq_len=256, dtype=bfloat16, no residual

Input Size: 4.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.051 0.98x

Standard Allreduce Rmsnorm Native Compiled 0.050 baseline

Flashinfer Fused Allreduce Rmsnorm Oneshot 0.064 0.77x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.050 1.00x

Configuration: seq_len=512, dtype=bfloat16, no residual

Input Size: 8.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.079 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.081 0.97x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.068 1.17x

Configuration: seq_len=1024, dtype=bfloat16, no residual

Input Size: 16.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.119 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.125 0.95x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.109 1.09x

Configuration: seq_len=2048, dtype=bfloat16, no residual

Input Size: 32.00 MB

Operation Time (ms) Speedup

Standard Allreduce Rmsnorm 0.195 1.00x

Standard Allreduce Rmsnorm Native Compiled 0.211 0.93x

Flashinfer Fused Allreduce Rmsnorm Twoshot 0.204 0.96x

@ilmarkov Is VLLM_ALLREDUCE_USE_SYMM_MEM=1 something that normal vLLM users would set by default? If it's good for performance, why can't we enable it by default? Does it require special environment or special builds? cc @ProExpertProg

@nvjullin Could you check if @ilmarkov 's measurements above match our understanding? Also, could you try if VLLM_ALLREDUCE_USE_SYMM_MEM=1 works in our case? Thanks!

Yes, it can be enabled by default. There is a PR for it. It works on Hopper and Blackwell.

Got it! we will try both your PRs and run some experiments on our side.

@ilmarkov Just to clarify: the PyTorch SYMM_MEM implementation does not support AR+Norm fusion, right? So only the AR part uses SYMM_MEM while Norm part is based on native PyT?

Yes, symm mem is only for allreduce part, Norm and quantization parts are in native pytorch.

nvpohanh · 2025-09-05T01:24:35Z

cc @nvjullin @elvischenv for vis

mergify · 2025-09-10T03:32:30Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ilmarkov.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

vllm/compilation/collective_fusion.py

nvpohanh · 2025-10-09T01:19:08Z

Hi @ilmarkov , is there any progress and ETA for this change? Thanks!

ilmarkov · 2025-10-09T15:03:53Z

Hi, @nvpohanh . @ProExpertProg works on general custom op matching in #24604. So we will apply allreduce related pattern matching after his PR is landed. I mark current PR as draft for now.

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ProExpertProg

LGTM, a few minor notes and then we can merge!

tests/compile/test_fusions_e2e.py

ProExpertProg · 2025-11-07T03:54:03Z

vllm/config/compilation.py

+
+    @staticmethod
+    def default_fi_allreduce_fusion_max_size_mb() -> dict[int, float]:
+        from vllm.compilation.collective_fusion import FI_ALLREDUCE_FUSION_MAX_SIZE_MB


@ilmarkov if this is still an issue to unblock we can just move this computation into the collective_fusion.py file. We can always move it back here later. I am also concerned that the head process (which shouldn't initialize CUDA) might initialize CUDA when querying device capability during device config (not sure if that happens in the head or just the workers).

But if stuff is working feel free to leave it as is

vllm/compilation/collective_fusion.py

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ilmarkov · 2025-11-10T18:59:47Z

We need to move dispatch and combine back under custom op as they conflict with torch.compile. In this PR we need only move (main experts) allreduce outside of custom op.

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ilmarkov · 2025-11-10T19:07:22Z

Validation.

vllm serve deepseek-ai/DeepSeek-V2-Lite --disable-log-requests --no-enable-prefix-caching -tp ${tp} -dp ${dp} --max-num-seqs 256 ${enable_expert_parallel} --port 8000 --compilation-config '{"pass_config":{"enable_fusion":false,"enable_attn_fusion":false,"enable_noop":true,"enable_sequen
ce_parallelism":false,"enable_async_tp":false,"enable_fi_allreduce_fusion":true,"fi_allreduce_fusion_max_size_mb":1}}'

TP=4

(model=deepseek-ai/DeepSeek-V2-Lite,base_url=http://0.0.0.0:8000/v1/completions,num_concurrent=50,max_retries=3,tokenized_requests=False), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3950|±  |0.0135|
|     |       |strict-match    |     5|exact_match|↑  |0.3927|±  |0.0135|

(DP+EP)=4

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3806|±  |0.0134|
|     |       |strict-match    |     5|exact_match|↑  |0.3783|±  |0.0134|

(TP+EP)=4

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.3798|±  |0.0134|
|     |       |strict-match    |     5|exact_match|↑  |0.3776|±  |0.0134|

Signed-off-by: ilmarkov <markovilya197@gmail.com>

nvpohanh · 2025-11-11T01:55:57Z

@ilmarkov Thanks for merging this! Just want to check: have we re-enabled symmetric memory communication by default? It was disabled in #26925 . Thanks!

Update: never mind. I just saw: #27671

zou3519

looks reasonable to me

zou3519 · 2025-11-17T19:34:41Z

vllm/compilation/collective_fusion.py

-        )
-        if use_flashinfer:
+
+        if num_tokens <= max_token_num:


@ilmarkov @ProExpertProg

@laithsakka and I ran over this a bit more offline and we're a bit worried this line might cause unintended specialization. Do you have instructions on how to trigger this line of code? (and if so, are you able to provide a tlparse of it? We want to check the symbolic shape constraints in the tlparse to see if this introduced anything negative)

@ilmarkov could you take a look? tlparse instructions here: https://docs.vllm.ai/en/latest/design/debug_vllm_compile

Btw, @zou3519 how do you want us to send you tlparse results, in an archive?

archive would work, maybe need a better way to share these...

Here are the tlparse results. But I am not sure if you will see the specialization in this line of code given that it is in custom op flashinfer_trtllm_fused_allreduce_norm
tl_out_vllm_fi_ar.tar.gz

… thresholds (#24248) Signed-off-by: Luka Govedič <lgovedic@redhat.com> Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: ilmarkov <markovilya197@gmail.com> Co-authored-by: Luka Govedič <lgovedic@redhat.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> (cherry picked from commit d17ecc6)

… thresholds (vllm-project#24248) Signed-off-by: Luka Govedič <lgovedic@redhat.com> Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: ilmarkov <markovilya197@gmail.com> Co-authored-by: Luka Govedič <lgovedic@redhat.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>

ilmarkov requested review from ProExpertProg, WoosukKwon, hmellor, houseroad, mgoin, robertgshaw2-redhat, simon-mo, tlrmchlsmth, yewentao256, youkaichao and zou3519 as code owners September 4, 2025 12:23

mergify bot added the performance Performance-related issues label Sep 4, 2025

mergify bot added the needs-rebase label Sep 4, 2025

ilmarkov mentioned this pull request Sep 4, 2025

[PERF] Allreduce Fusion tuning and compile_ranges introduction #22086

Closed

gemini-code-assist bot reviewed Sep 4, 2025

View reviewed changes

nvpohanh reviewed Sep 5, 2025

View reviewed changes

vllm/compilation/collective_fusion.py Outdated Show resolved Hide resolved

nvpohanh reviewed Sep 5, 2025

View reviewed changes

ilmarkov force-pushed the imarkov/fused_allreduce_torch_native branch from e808818 to 61ebc95 Compare September 8, 2025 12:02

mergify bot removed the needs-rebase label Sep 8, 2025

mergify bot added the needs-rebase label Sep 10, 2025

ProExpertProg reviewed Sep 12, 2025

View reviewed changes

vllm/compilation/collective_fusion.py Outdated Show resolved Hide resolved

ProExpertProg mentioned this pull request Sep 25, 2025

[Feature]: Enabling performance optimizations by default #25689

Open

11 tasks

ProExpertProg added this to the vllm==v0.12.0/torch==2.9.0 compilation improvements milestone Sep 25, 2025

ilmarkov marked this pull request as draft October 9, 2025 15:03

ilmarkov added 2 commits November 6, 2025 10:19

Update comments

71c6b72

Signed-off-by: ilmarkov <markovilya197@gmail.com>

Merge branch 'main' into imarkov/fused_allreduce_torch_native

56273da

ProExpertProg approved these changes Nov 7, 2025

View reviewed changes

Minor updates

2785e4d

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ilmarkov mentioned this pull request Nov 7, 2025

[Compile] Conditional compilation. Introduce compile_ranges #24252

Merged

3 tasks

ProExpertProg and others added 2 commits November 7, 2025 15:48

Merge branch 'main' into imarkov/fused_allreduce_torch_native

1f83a66

Merge branch 'main' into imarkov/fused_allreduce_torch_native

ab33605

ProExpertProg approved these changes Nov 10, 2025

View reviewed changes

ilmarkov added 2 commits November 10, 2025 16:16

Merge branch 'main' into imarkov/fused_allreduce_torch_native

3fac39b

Fix SP

b0a3884

Signed-off-by: ilmarkov <markovilya197@gmail.com>

Move dispatch+combine under custom_op

90afc4e

Signed-off-by: ilmarkov <markovilya197@gmail.com>

fix

0e453e4

Signed-off-by: ilmarkov <markovilya197@gmail.com>

ProExpertProg merged commit d17ecc6 into vllm-project:main Nov 10, 2025
54 checks passed

github-project-automation bot moved this from In review to Done in torch.compile integration Nov 10, 2025

robertgshaw2-redhat mentioned this pull request Nov 11, 2025

[CI Failure]: Blackwell Fusion E2E Tests #28468

Closed

3 tasks

zou3519 reviewed Nov 11, 2025

View reviewed changes

zou3519 mentioned this pull request Nov 17, 2025

[Bug]: When compiling with ranges, we should pass the range information to Inductor #28868

Open

1 task

zou3519 reviewed Nov 17, 2025

View reviewed changes

maxyanghu mentioned this pull request Nov 19, 2025

[Bug]: Qwen3-VL-235B-A22B-Instruct-NVFP4 fused moe failed to be traced by torch dynamo #29015

Closed

1 task

nvjullin mentioned this pull request Nov 21, 2025

Removed unused env var VLLM_FLASHINFER_ALLREDUCE_FUSION_THRESHOLDS_MB #29152

Open

5 tasks

elvischenv mentioned this pull request Nov 21, 2025

[Bug]: Got additional triton kernel when using native rms_norm op matching with all_reduce+rms_norm fusion #29181

Closed

nvjullin mentioned this pull request Dec 15, 2025

[Misc] Moved override for allreduce fusion thresholds from env var to config #23722

Closed

5 tasks

ilmarkov deleted the imarkov/fused_allreduce_torch_native branch December 15, 2025 13:25

mayank-ketkar-sf mentioned this pull request Feb 17, 2026

[Bugfix] Fix benchmark_fused_collective crash on CustomOp init #34665

Merged

3 tasks

	return allreduce[3], allreduce[1]
	return allreduce[2], allreduce[0]

	return allreduce[1], allreduce[2]
	return allreduce[0], allreduce[1]

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.029	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.030	0.99x
Flashinfer Fused Allreduce Rmsnorm Oneshot	0.012	2.39x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.086	0.34x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.023	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.024	0.99x
Flashinfer Fused Allreduce Rmsnorm Oneshot	0.033	0.71x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.052	0.45x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.031	0.97x
Standard Allreduce Rmsnorm Native Compiled	0.030	baseline
Flashinfer Fused Allreduce Rmsnorm Oneshot	0.064	0.47x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.050	0.60x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.044	0.98x
Standard Allreduce Rmsnorm Native Compiled	0.043	baseline
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.297	0.15x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.071	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.077	0.93x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.109	0.66x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.135	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.143	0.94x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.205	0.66x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.032	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.032	1.00x
Flashinfer Fused Allreduce Rmsnorm Oneshot	0.033	0.97x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.052	0.62x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.051	0.98x
Standard Allreduce Rmsnorm Native Compiled	0.050	baseline
Flashinfer Fused Allreduce Rmsnorm Oneshot	0.064	0.77x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.050	1.00x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.079	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.081	0.97x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.068	1.17x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.119	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.125	0.95x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.109	1.09x

Operation	Time (ms)	Speedup
Standard Allreduce Rmsnorm	0.195	1.00x
Standard Allreduce Rmsnorm Native Compiled	0.211	0.93x
Flashinfer Fused Allreduce Rmsnorm Twoshot	0.204	0.96x

Uh oh!

Conversation

ilmarkov commented Sep 4, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Uh oh!

mergify bot commented Sep 4, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Symm mem enabled

Configuration: seq_len=32, dtype=bfloat16, no residual

Configuration: seq_len=64, dtype=bfloat16, no residual

Configuration: seq_len=128, dtype=bfloat16, no residual

Configuration: seq_len=256, dtype=bfloat16, no residual

Configuration: seq_len=256, dtype=bfloat16, no residual

Configuration: seq_len=512, dtype=bfloat16, no residual

Configuration: seq_len=1024, dtype=bfloat16, no residual

Configuration: seq_len=2048, dtype=bfloat16, no residual

Default allreduce

Configuration: seq_len=32, dtype=bfloat16, no residual

Configuration: seq_len=64, dtype=bfloat16, no residual

Configuration: seq_len=128, dtype=bfloat16, no residual

Configuration: seq_len=256, dtype=bfloat16, no residual

Configuration: seq_len=512, dtype=bfloat16, no residual

Configuration: seq_len=1024, dtype=bfloat16, no residual

Configuration: seq_len=2048, dtype=bfloat16, no residual

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilmarkov Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvpohanh commented Sep 5, 2025

Uh oh!

mergify bot commented Sep 10, 2025

Uh oh!

Uh oh!

nvpohanh commented Oct 9, 2025

Uh oh!

ilmarkov commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ilmarkov commented Nov 10, 2025

Uh oh!

ilmarkov commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nvpohanh commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ilmarkov commented Sep 4, 2025 •

edited by github-actions bot

Loading

ilmarkov Sep 5, 2025 •

edited

Loading

ilmarkov commented Oct 9, 2025 •

edited

Loading

ilmarkov commented Nov 10, 2025 •

edited

Loading

nvpohanh commented Nov 11, 2025 •

edited

Loading

zou3519 Nov 19, 2025 •

edited

Loading

ilmarkov Nov 19, 2025 •

edited

Loading