Skip to content

[CPU] Add Qwen3.5 model optimization for CPU#19484

Merged
Kangyan-Zhou merged 43 commits intosgl-project:mainfrom
jianan-gu:qwen3.5_cpu_opt
Apr 26, 2026
Merged

[CPU] Add Qwen3.5 model optimization for CPU#19484
Kangyan-Zhou merged 43 commits intosgl-project:mainfrom
jianan-gu:qwen3.5_cpu_opt

Conversation

@jianan-gu
Copy link
Copy Markdown
Contributor

@jianan-gu jianan-gu commented Feb 27, 2026 •
edited
Loading

This PR (work with @blzheng ) adds support for Qwen3.5 series with cpu optimized performance, including changes:

  1. Dtype support in fusion of fused_sigmoid_gating_delta_rule_update
  2. Continues support for fusion of fused_qkvzba_split_reshape_cat_contiguous_cpu
  3. TP cases padding support for both bf16 and fp8
  4. Refinements for logging CPU padding logic.

Note that this PR depends on previous CPU mrope kernel support #12531

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Feb 27, 2026

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added Multi-modal multi-modal language model sgl-kernel labels Feb 27, 2026
@jianan-gu jianan-gu changed the title [CPU] Add Qwen3.5 model optimized opt [CPU] Add Qwen3.5 model optimization for CPU Feb 27, 2026
jianan-gu and others added 4 commits March 2, 2026 02:28
@jianan-gu
[PR19484] fix tp head dim
9f81037
@blzheng @jianan-gu
[PR19484] Add logging info for config updates (#104)
ea4c197 
* refactor and format code

* add log for config setting and updating

* only log config update info on rank 0

* refactor code

change log info to debug
@blzheng @jianan-gu
apply sgl kernel when tie_word_embeddings is True (#105)
76b5980
@blzheng @jianan-gu
[PR to track] reduce duplicated copy overhead in moe (#106)
37d4b71
@jianan-gu jianan-gu marked this pull request as ready for review March 3, 2026 01:43
@mingfeima
Copy link
Copy Markdown
Collaborator

mingfeima commented Apr 22, 2026

@jianan-gu fail at https://github.com/sgl-project/sglang/actions/runs/24720109083/job/72306535388?pr=19484#step:6:2541. rebase and test again.

JustinTong0323
JustinTong0323 reviewed Apr 22, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@JustinTong0323 JustinTong0323 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few issues in the follow-up — the three new asserts use assert (cond, msg,) which Python parses as asserting a non-empty tuple, so they never fire (also triggers SyntaxWarning: assertion is always true). And the two new TORCH_CHECK lines in qwen3.cpp are missing trailing ;.

Comment thread python/sglang/srt/model_loader/weight_utils.py Outdated
Comment thread python/sglang/srt/models/qwen3_5.py Outdated
Comment thread python/sglang/srt/layers/attention/mamba/mamba.py Outdated
Comment thread sgl-kernel/csrc/cpu/model/qwen3.cpp Outdated
Comment thread python/sglang/srt/layers/attention/mamba/mamba.py Outdated
Comment thread python/sglang/srt/utils/common.py Outdated
Comment thread python/sglang/srt/configs/update_config.py Outdated
@JustinTong0323 JustinTong0323 self-assigned this Apr 24, 2026
JustinTong0323
JustinTong0323 reviewed Apr 24, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@JustinTong0323 JustinTong0323 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round-2 fixes all landed cleanly — asserts, TORCH_CHECK, head_dim resolver, rank-0 logging all look good. Two new things surfaced on a final pass, one of which is a crash on Qwen3-Omni with unaligned TP.

Comment thread python/sglang/srt/configs/update_config.py Outdated
Comment thread python/sglang/srt/utils/common.py
Comment thread python/sglang/srt/layers/attention/mamba/mamba.py
@JustinTong0323
Copy link
Copy Markdown
Collaborator

JustinTong0323 commented Apr 24, 2026

/rerun-failed-ci

5 similar comments
@jianan-gu
Copy link
Copy Markdown
Contributor Author

jianan-gu commented Apr 24, 2026

/rerun-failed-ci

@jianan-gu
Copy link
Copy Markdown
Contributor Author

jianan-gu commented Apr 25, 2026

/rerun-failed-ci

@jianan-gu
Copy link
Copy Markdown
Contributor Author

jianan-gu commented Apr 25, 2026

/rerun-failed-ci

@jianan-gu
Copy link
Copy Markdown
Contributor Author

jianan-gu commented Apr 26, 2026

/rerun-failed-ci

@jianan-gu
Copy link
Copy Markdown
Contributor Author

jianan-gu commented Apr 26, 2026

/rerun-failed-ci

@Kangyan-Zhou Kangyan-Zhou merged commit 10fd0fa into sgl-project:main Apr 26, 2026
476 of 547 checks passed
vguduruTT pushed a commit to vguduruTT/sglang that referenced this pull request May 2, 2026
@jianan-gu @blzheng
@mingfeima @JustinTong0323
[CPU] Add Qwen3.5 model optimization for CPU (sgl-project#19484)
c96f98d 
Co-authored-by: Zheng, Beilei <beilei.zheng@intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mingfeima mingfeima mingfeima approved these changes

@mickqian mickqian mickqian approved these changes

@JustinTong0323 JustinTong0323 JustinTong0323 approved these changes

@zhyncs zhyncs Awaiting requested review from zhyncs

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg FlamingoPg is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@hebiao064 hebiao064 Awaiting requested review from hebiao064 hebiao064 is a code owner

@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner

@Qiaolin-Yu Qiaolin-Yu Awaiting requested review from Qiaolin-Yu

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner

@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner

@yuan-luo yuan-luo Awaiting requested review from yuan-luo yuan-luo is a code owner

@yhyang201 yhyang201 Awaiting requested review from yhyang201 yhyang201 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

+1 more reviewer

@blzheng blzheng blzheng left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@JustinTong0323 JustinTong0323

Labels

cpu cpu backend performance optimization intel Multi-modal multi-modal language model run-ci sgl-kernel

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@jianan-gu @mingfeima @JustinTong0323 @mickqian @blzheng @Kangyan-Zhou