Double vision prefill throughput by defaulting to optimal vision attention backend by AlienKevin · Pull Request #8484 · sgl-project/sglang

AlienKevin · 2025-07-29T00:34:23Z

Motivation

TLDR; Vision attention defaults to inefficient SDPA even on platforms that support more efficient Triton backend. By defaulting to most optimal supported attention backend, we can double vision perfill throughput on VLMs like Qwen 2.5 VL 7B on H200.
See #8179 (comment)

Benchmark & Profiling

Profiling Qwen 2.5 VL 7B before and after PR on GB200 shows 1.6x increase in throughput under ISL=1000, OSL=2, and concurrency=256.
Before PR:

"Input Tokens/s": 7407.142234200068,
"Avg TTFT(s)": 20.554337582262974,
"Avg TPOT(s)": 14.229874295137847

After PR:

"Input Tokens/s": 12073.272607543215,
"Avg TTFT(s)": 12.099749442036076,
"Avg TPOT(s)": 8.979351270686001

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

…ntion backend

gemini-code-assist

Summary of Changes

Hello @AlienKevin, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've implemented a change to significantly improve the vision prefill throughput by ensuring that the most optimal attention backend is used by default. Previously, vision attention would often default to the less efficient SDPA even when a more performant Triton backend was available. My changes now detect CUDA availability and prioritize triton_attn, which has been shown to double vision prefill throughput on VLMs like Qwen 2.5 VL 7B on H200. This optimization leads to a substantial performance gain, as evidenced by benchmarks showing a 1.6x increase in input tokens per second.

Highlights

Optimized Vision Attention Backend Selection: The system now intelligently defaults to triton_attn on CUDA-enabled platforms for vision attention, falling back to sdpa otherwise, significantly boosting performance.
Improved Qwen 2.5 VL Model Integration: The Qwen 2.5 VL model's attention implementation initialization has been updated to leverage this new dynamic backend selection, removing its previous hardcoded sdpa preference.
Significant Throughput Gains: Benchmarks demonstrate a 1.6x increase in input tokens/s, leading to faster Time To First Token (TTFT) and Time Per Output Token (TPOT) for vision models.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request improves vision prefill throughput by defaulting to the Triton attention backend on CUDA-enabled platforms, which is more efficient than the previous SDPA default. The changes are logical and well-supported by the provided benchmarks. The implementation correctly modifies VisionAttention to select the optimal backend and updates Qwen2_5_VisionBlock to allow this new default behavior. I have one minor style suggestion to improve code quality.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

gemini-code-assist · 2025-07-29T00:43:50Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>

JustinTong0323 · 2025-07-29T01:02:55Z

Thanks for the contribution! May I ask why we choose triton_attn instead of fa3?

AlienKevin · 2025-07-29T02:02:53Z

Thanks for the contribution! May I ask why we choose triton_attn instead of fa3?

I found them to have similar performance but triton_attn is supported on more devices, like GB200.

AlienKevin · 2025-07-31T16:44:34Z

@JustinTong0323 Just checking in, anything else blocking this PR to be merged? Thanks.

JustinTong0323 · 2025-08-01T04:53:23Z

@JustinTong0323 Just checking in, anything else blocking this PR to be merged? Thanks.

ping me in slack if CI passed, or need to rerun, thanks

…on to reduce false negatives

JustinTong0323

Thanks Kevin!

…ntion backend (sgl-project#8484) Co-authored-by: Xiang (Kevin) Li <lik@nvidia.com>

Double vision prefill throughput by defaulting to optimal vision atte…

751c001

…ntion backend

AlienKevin requested review from BBuf, HaiShaw, Ying1123, ch-wan, ispobock, merrymercy and zhyncs as code owners July 29, 2025 00:34

AlienKevin requested a review from a team July 29, 2025 00:34

gemini-code-assist Bot reviewed Jul 29, 2025

View reviewed changes

Comment thread python/sglang/srt/models/qwen2_5_vl.py Outdated

AlienKevin mentioned this pull request Jul 29, 2025

[Perf] Profiling and Monitoring VLM performance #8179

Closed

2 tasks

JustinTong0323 and others added 2 commits July 28, 2025 17:43

Merge branch 'main' into optimized-vision-attn

15285b2

Update python/sglang/srt/models/qwen2_5_vl.py

f2c0564

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fix lint

0ac9f7a

Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>

JustinTong0323 requested a review from kushanam as a code owner July 29, 2025 00:59

Merge branch 'main' into optimized-vision-attn

d380765

JustinTong0323 added 2 commits July 31, 2025 10:08

Merge branch 'main' into optimized-vision-attn

b14c187

Merge branch 'main' into optimized-vision-attn

07ffd7d

JustinTong0323 and others added 6 commits August 4, 2025 12:29

Merge branch 'main' into optimized-vision-attn

bbfdda2

Handle cu_seqlens=None in VisionTritonAttention

1b559a3

Merge branch 'main' into optimized-vision-attn

aa6efe0

Relax test_video_images_chat_completion and _test_video_chat_completi…

80af006

…on to reduce false negatives

Merge branch 'main' into optimized-vision-attn

2926131

Merge branch 'main' into optimized-vision-attn

2c8f13f

Merge branch 'main' into optimized-vision-attn

4a98d99

JustinTong0323 requested a review from Edwardf0t1 as a code owner August 8, 2025 23:14

Merge branch 'main' into optimized-vision-attn

c4ae89e

JustinTong0323 approved these changes Aug 12, 2025

View reviewed changes

JustinTong0323 added the ready-to-merge The PR is ready to merge after the CI is green. label Aug 12, 2025

zhyncs merged commit 3b3b3ba into sgl-project:main Aug 13, 2025
179 of 192 checks passed

narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025

Double vision prefill throughput by defaulting to optimal vision atte…

d1c6739

…ntion backend (sgl-project#8484) Co-authored-by: Xiang (Kevin) Li <lik@nvidia.com>

AlienKevin mentioned this pull request Sep 5, 2025

Fast rotary embedding mickqian/sglang#7

Closed

MahmoudAshraf97 pushed a commit to MahmoudAshraf97/sglang that referenced this pull request Sep 8, 2025

Double vision prefill throughput by defaulting to optimal vision atte…

9b438f2

…ntion backend (sgl-project#8484) Co-authored-by: Xiang (Kevin) Li <lik@nvidia.com>

AlienKevin mentioned this pull request Sep 16, 2025

Fast rotary embedding #10527

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Double vision prefill throughput by defaulting to optimal vision attention backend#8484

Double vision prefill throughput by defaulting to optimal vision attention backend#8484
zhyncs merged 15 commits intosgl-project:mainfrom
AlienKevin:optimized-vision-attn

AlienKevin commented Jul 29, 2025 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

gemini-code-assist Bot commented Jul 29, 2025

Uh oh!

JustinTong0323 commented Jul 29, 2025

Uh oh!

AlienKevin commented Jul 29, 2025

Uh oh!

AlienKevin commented Jul 31, 2025

Uh oh!

JustinTong0323 commented Aug 1, 2025

Uh oh!

JustinTong0323 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

AlienKevin commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Benchmark & Profiling

Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot commented Jul 29, 2025

Uh oh!

JustinTong0323 commented Jul 29, 2025

Uh oh!

AlienKevin commented Jul 29, 2025

Uh oh!

AlienKevin commented Jul 31, 2025

Uh oh!

JustinTong0323 commented Aug 1, 2025

Uh oh!

JustinTong0323 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlienKevin commented Jul 29, 2025 •

edited

Loading