Skip to content

Update Flashinfer to 0.2.14.post1#23537

Merged
simon-mo merged 13 commits intovllm-project:mainfrom
weireweire:flashinfer-update
Aug 26, 2025
Merged

Update Flashinfer to 0.2.14.post1#23537
simon-mo merged 13 commits intovllm-project:mainfrom
weireweire:flashinfer-update

Conversation

@weireweire
Copy link
Copy Markdown
Contributor

@weireweire weireweire commented Aug 25, 2025 •
edited by github-actions Bot
Loading

Purpose

Update flashinfer to

Test Plan

lm_eval on llama3

gpt-oss/eval test on gpt-oss

  • python3 -m gpt_oss.evals --sampler chat_completions --model gpt-oss-120b --reasoning-effort low,medium --n-threads 512 --eval gpqa

Test Result

llama3 FP8 tp2

Tasks Version Filter n-shot Metric Value Stderr
gsm8k 3 flexible-extract 5 exact_match ↑ 0.938 ± 0.0108
strict-match 5 exact_match ↑ 0.904 ± 0.0132

llama3 FP4 tp1

Tasks Version Filter n-shot Metric Value Stderr
gsm8k 3 flexible-extract 5 exact_match ↑ 0.930 ± 0.0114
strict-match 5 exact_match ↑ 0.842 ± 0.0163

gpt-oss tp1:
[{'eval_name': 'gpqa', 'model_name': 'gpt-oss-120b-low_temp1.0_20250825_021150', 'metric': 0.6414141414141414}, {'eval_name': 'gpqa', 'model_name': 'gpt-oss-120b-medium_temp1.0_20250825_021150', 'metric': 0.711489898989899}]

(Optional) Documentation Update

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

IwakuraRein and others added 11 commits August 25, 2025 06:24
@IwakuraRein @weireweire
fix after flashinfer autotuner
82de5e0 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
add warmup
bfc1f20 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
address comment
4cd9551 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
Update flashinfer tag
57e4e21 
Signed-off-by: siyuanf <siyuanf@nvidia.com>
@IwakuraRein @weireweire
address comment
8764384 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
address comment
e9f6e12 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
update dockerfile
31bac6a 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
address todo
74c5e7b 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@IwakuraRein @weireweire
address todo
e635283 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
@weireweire
Fix flashinfer swizzle enum name.
c0a5e4b 
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
@weireweire
update flashinfer to 0.2.14.post1
d522d9a 
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Aug 25, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates Flashinfer to version 0.2.14.post1, which addresses a performance issue in the allreduce fusion kernel and incorporates API changes. The changes also include enabling FlashInfer autotuning before CUDA graph capture for better performance. My review found a critical typo in a variable name (max_captute_size instead of max_capture_size) in mxfp4.py which would lead to a runtime error. I've provided suggestions to fix this.

Comment thread vllm/model_executor/layers/quantization/mxfp4.py Outdated
Comment thread vllm/model_executor/layers/quantization/mxfp4.py Outdated
@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 25, 2025
mgoin and others added 2 commits August 25, 2025 11:08
@mgoin @gemini-code-assist
Apply suggestion from @gemini-code-assist[bot]
7313a92 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
@mgoin @gemini-code-assist
Apply suggestion from @gemini-code-assist[bot]
8548148 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
yewentao256
yewentao256 reviewed Aug 25, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@yewentao256 yewentao256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!
Could you also report vllm bench metric results so that we can see if we have some improvement for E2E throughput?

mgoin
mgoin approved these changes Aug 25, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for putting everything together. Let's see the CI

@IwakuraRein IwakuraRein mentioned this pull request Aug 25, 2025
Closed
4 tasks
@simon-mo simon-mo merged commit ae06788 into vllm-project:main Aug 26, 2025
64 of 70 checks passed
@weireweire weireweire mentioned this pull request Aug 26, 2025
Closed
4 tasks
tc-mb pushed a commit to tc-mb/vllm that referenced this pull request Aug 27, 2025
@weireweire @IwakuraRein
@mgoin @gemini-code-assist @tc-mb
Update Flashinfer to 0.2.14.post1 (vllm-project#23537)
f5bc341 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: siyuanf <siyuanf@nvidia.com>
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: tc-mb <caitianchi@modelbest.cn>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@weireweire @IwakuraRein
@mgoin @gemini-code-assist @epwalsh
Update Flashinfer to 0.2.14.post1 (vllm-project#23537)
e9c8152 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: siyuanf <siyuanf@nvidia.com>
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
xiao-llm pushed a commit to xiao-llm/vllm that referenced this pull request Aug 28, 2025
@weireweire @IwakuraRein
@mgoin @gemini-code-assist @amd-xiaoyu12
Update Flashinfer to 0.2.14.post1 (vllm-project#23537)
e699971 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: siyuanf <siyuanf@nvidia.com>
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Xiao Yu <xiao.yu@amd.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
@weireweire @IwakuraRein
@mgoin @gemini-code-assist @zhewenl
Update Flashinfer to 0.2.14.post1 (vllm-project#23537)
a6b8ab6 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: siyuanf <siyuanf@nvidia.com>
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Sep 3, 2025
@weireweire @IwakuraRein
@mgoin @gemini-code-assist @zhewenl
Update Flashinfer to 0.2.14.post1 (vllm-project#23537)
a40904e 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: siyuanf <siyuanf@nvidia.com>
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
ABC12345anouys pushed a commit to ABC12345anouys/vllm that referenced this pull request Sep 25, 2025
@weireweire @IwakuraRein
@mgoin @gemini-code-assist @ABC12345anouys
Update Flashinfer to 0.2.14.post1 (vllm-project#23537)
d4569c4 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: siyuanf <siyuanf@nvidia.com>
Signed-off-by: Weiliang Liu <weiliangl@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yewentao256 yewentao256 yewentao256 left review comments

@mgoin mgoin mgoin approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96

@comaniac comaniac Awaiting requested review from comaniac

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

@zou3519 zou3519 Awaiting requested review from zou3519 zou3519 is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@weireweire @mgoin @yewentao256 @simon-mo @IwakuraRein