Update model and feature support for Ascend NPU by Hexq0210 · Pull Request #16005 · sgl-project/sglang

Hexq0210 · 2025-12-28T08:51:37Z

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.
Work with maintainers to merge your PR. See the PR Merge Process

gemini-code-assist · 2025-12-28T08:51:54Z

Summary of Changes

Hello @Hexq0210, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands and clarifies the documentation for SGLang's support on Ascend NPU. It provides users with more comprehensive guidance on setting up their environment, including detailed Docker instructions and critical environment variable configurations. Furthermore, it introduces dedicated sections outlining the specific models and features supported, ensuring users can easily identify compatible configurations for various LLM and Multimodal LLM applications.

Highlights

Enhanced Documentation for Ascend NPU Setup: The ascend_npu.md file has been significantly updated to provide more detailed instructions for preparing the running environment, including Docker image acquisition methods (download from DockerHub/Quay.io or build from Dockerfile) and critical environment variable settings like HF_TOKEN and SGLANG_SET_CPU_AFFINITY.
New Section for Running SGLang Services: A dedicated section has been added to ascend_npu.md with comprehensive command examples for launching SGLang services. This includes configurations for both Large Language Models (covering PD Mixed and PD Separation scenarios with separate prefill, decode, and router launches) and Multimodal Language Models (PD Mixed Scene).
Comprehensive Model Support Listing: Two new documentation files, ascend_npu_support_models.md and ascend_npu_support_features.md, have been introduced. The ascend_npu_support_models.md file provides detailed tables of supported Large Language Models, Multimodal Language Models, Embedding Models, Reward Models, and Rerank Models, indicating their compatibility with Ascend NPU A2 and A3 platforms.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request significantly enhances the documentation for Ascend NPU support by adding detailed instructions for environment setup, Docker usage, and running SGLang services. It also introduces new pages listing supported models and features. My review focuses on improving the clarity and formatting of the new documentation. I've suggested changes to make the Docker image instructions more user-friendly, improve table readability for long model lists, and correct formatting inconsistencies caused by stray tab characters. Overall, these are valuable additions for users on the Ascend platform.

gemini-code-assist · 2025-12-28T08:53:54Z

+# You can choose between dockerhub and quay.io
+dockerhub: docker.io/lmsysorg/sglang:$tag
+quay.io: quay.io/ascend/sglang:$tag
+# Main-based tag, change main to specific version like v0.5.6,
+# you can get image for specific version
+Atlas A3 server : {main}-cann8.3.rc2-a3
+Atlas A2 server: {main}-cann8.3.rc2-910b


This code block is confusing as it mixes instructions with non-executable lines inside a shell block. To improve clarity, it's better to provide commented-out, ready-to-use docker pull examples.

Suggested change

# You can choose between dockerhub and quay.io

dockerhub: docker.io/lmsysorg/sglang:$tag

quay.io: quay.io/ascend/sglang:$tag

# Main-based tag, change main to specific version like v0.5.6,

# you can get image for specific version

Atlas A3 server : {main}-cann8.3.rc2-a3

Atlas A2 server: {main}-cann8.3.rc2-910b

# You can choose between Docker Hub (docker.io/lmsysorg/sglang) and Quay.io (quay.io/ascend/sglang).

# The tag is constructed as {version}-cann8.3.rc2-{server_type}.

# Replace {version} with `main` for the latest development version, or a specific version like `v0.5.6`.

# Example for Atlas A3 server:

# docker pull docker.io/lmsysorg/sglang:main-cann8.3.rc2-a3

# Example for Atlas A2 server:

# docker pull docker.io/lmsysorg/sglang:main-cann8.3.rc2-910b

gemini-code-assist · 2025-12-28T08:53:55Z

+|--------------------------------|------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| DeepSeek                       | DeepSeek V1, V2, V3(V3.1,V3.2), R1                                                                               | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
+| Qwen                           | Qwen 3, Qwen 3Moe                                                                                                | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
+| Llama                          | meta-llama/Llama-4-Scout-17B-16E-Instruct,AI-ModelScope/Llama-3.1-8B-Instruct,LLM-Research/Llama-3.2-1B-Instruct |  **<span style="color: red;">×</span>**  |  **<span style="color: red;">×</span>**  |


This cell contains a long, comma-separated list of model names, which makes the table very wide and hard to read. Consider using   tags to place each model on a new line for better readability.

Suggested change

| Llama | meta-llama/Llama-4-Scout-17B-16E-Instruct,AI-ModelScope/Llama-3.1-8B-Instruct,LLM-Research/Llama-3.2-1B-Instruct | **×** | **×** |

| Llama | meta-llama/Llama-4-Scout-17B-16E-Instruct AI-ModelScope/Llama-3.1-8B-Instruct LLM-Research/Llama-3.2-1B-Instruct | **×** | **×** |

gemini-code-assist · 2025-12-28T08:53:55Z

+| Qwen-VL (Qwen2 series)         | 	Qwen/Qwen3-VL-235B-A22B-Instruct              | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| DeepSeek-VL2                   | deepseek-ai/deepseek-vl2                       | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| Janus-Pro (1B, 7B)             | 	deepseek-ai/Janus-Pro-7B                      | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
+| MiniCPM-V / MiniCPM-o          | 	openbmb/MiniCPM-V-2_6                         | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| Gemma 3 (Multimodal)           | 	google/gemma-3-4b-it                          | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
+| Mistral-Small-3.1-24B          | 	mistralai/Mistral-Small-3.1-24B-Instruct-2503 | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| Phi-4-multimodal-instruct      | 	microsoft/Phi-4-multimodal-instruct           | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| MiMo-VL (7B)                   | 	XiaomiMiMo/MiMo-VL-7B-RL                      | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| LLaVA (v1.5 & v1.6)            | 	AI-ModelScope/llava-v1.6-34b                  | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
+| LLaVA-NeXT (8B, 72B)           | 	lmms-lab/llava-next-72b                       | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
+| LLaVA-OneVision                | 	lmms-lab/llava-onevision-qwen2-7b-ov          | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| Kimi-VL (A3B)                  | 	Kimi/Kimi-VL-A3B-Instruct                     | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+| GLM-4.5V (106B) / GLM-4.1V(9B) | ZhipuAI/GLM-4.5V                               | **<span style="color: red;">×</span>**   | **<span style="color: green;">√</span>** |
+| Llama 3.2 Vision (11B)         | meta-llama/Llama-3.2-11B-Vision-Instruct       | **<span style="color: red;">×</span>**   |  **<span style="color: red;">×</span>**  |
+
+## Embedding Models
+
+| Model Family             | Example Models                            | A2 Supported                           |               A3 Supported               |
+|--------------------------|-------------------------------------------|----------------------------------------|:----------------------------------------:|
+| E5 (Llama/Mistral based) | 	intfloat/e5-mistral-7b-instruct          | **<span style="color: red;">×</span>** |  **<span style="color: red;">×</span>**  |
+| GTE-Qwen2                | 	iic/gte_Qwen2-1.5B-instruct              | **<span style="color: red;">×</span>** |  **<span style="color: red;">×</span>**  |
+| Qwen3-Embedding          | 	Qwen/Qwen3-Embedding-8B                  | **<span style="color: red;">×</span>** |  **<span style="color: red;">×</span>**  |
+| GME (Multimodal)         | 	Alibaba-NLP/gme-Qwen2-VL-2B-Instruct     | **<span style="color: red;">×</span>** |  **<span style="color: red;">×</span>**  |
+| CLIP                     | 	AI-ModelScope/clip-vit-large-patch14-336 | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
+| BGE                      | 	BAAI/bge-large-en-v1.5                   | **<span style="color: red;">×</span>** |  **<span style="color: red;">×</span>**  |
+
+## Reward Models
+
+| Model Family              | Example Models                              | A2 Supported                           |               A3 Supported               |
+|---------------------------|---------------------------------------------|----------------------------------------|:----------------------------------------:|
+| Llama3.1 Reward           | 	Skywork/Skywork-Reward-Llama-3.1-8B-v0.2   | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
+| InternLM 2 Reward         | 	Shanghai_AI_Laboratory/internlm2-7b-reward | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
+| Qwen2.5 Reward - Math     | 	Qwen/Qwen2.5-Math-RM-72B                   | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
+| Qwen2.5 Reward - Sequence | 	jason9693/Qwen2.5-1.5B-apeach              | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
+| Gemma 2-27B Reward        | 	Skywork/Skywork-Reward-Gemma-2-27B-v0.2    | **<span style="color: red;">×</span>** |  **<span style="color: red;">×</span>**  |


The model names in the 'Multimodal Language Models', 'Embedding Models', and 'Reward Models' tables have a leading tab character (\t), which causes inconsistent formatting. Please remove these tabs for better readability. This affects lines 43-56, 62-67, and 73-77.

gemini-code-assist · 2025-12-28T08:53:55Z

+
+| Model Family  | Example Models          |              A2 Supported              |              A3 Supported              |
+|---------------|-------------------------|:--------------------------------------:|:--------------------------------------:|
+| BGE-Reranker	 | BAAI/bge-reranker-v2-m3 | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |


The model family name BGE-Reranker has a trailing tab character (\t). Please remove it for consistent formatting.

Suggested change

| BGE-Reranker | BAAI/bge-reranker-v2-m3 | **×** | **×** |

| BGE-Reranker | BAAI/bge-reranker-v2-m3 | **×** | **×** |

…glang into eagle-sche * 'ifmn/eagle-dp-attn' of https://github.com/sgl-project/sglang: (22 commits) dp scheduler enhance support with chunked prefill (sgl-project#16071) modify suffix decoding CI dependency update (sgl-project#16063) fix rotary_embedding init npu (sgl-project#16011) feat: bugfix and accuracy fix for stablelm2_1_6b (sgl-project#15932) Update model and feature support for Ascend NPU (sgl-project#16005) Bugfix for Llama4 (sgl-project#15929) Bugfix for ds-vl2 (sgl-project#15894) gme qwen vl runners fix (sgl-project#15899) add profiling in scheduler (sgl-project#15876) llama use triton rope op (sgl-project#15855) suffix decoding adapt npu suffix decoding adapt npu Add suffix decoding speculative algorithm from feature 13553 cherry sgl-project#15434: qwen3 vl performance update cherry sgl-project#15597: fix Qwen3-VL-30B-A3B-Instruct accuracy loss [Schedule] bug fix for schedule enhancer (sgl-project#15834) minilb support roundrobin (sgl-project#15824) fix torchair compile issue cherry sgl-project#15187: lora fix ... # Conflicts: # python/sglang/srt/managers/scheduler.py # python/sglang/srt/managers/scheduler_enhancer.py

Update model and feature support for Ascend NPU

5b1bfd3

github-actions Bot added documentation Improvements or additions to documentation npu labels Dec 28, 2025

gemini-code-assist Bot reviewed Dec 28, 2025

View reviewed changes

Update model and feature support for Ascend NPU

ea5ed91

jiashaokun-1 reviewed Dec 28, 2025

View reviewed changes

Comment thread docs/platforms/ascend_npu.md

jiashaokun-1 reviewed Dec 28, 2025

View reviewed changes

Comment thread docs/platforms/ascend_npu.md

Update model and feature support for Ascend NPU

f133298

iforgetmyname merged commit e15910f into sgl-project:ifmn/eagle-dp-attn Dec 28, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update model and feature support for Ascend NPU#16005

Update model and feature support for Ascend NPU#16005
iforgetmyname merged 3 commits intosgl-project:ifmn/eagle-dp-attnfrom
Hexq0210:doc_for_business

Hexq0210 commented Dec 28, 2025

Uh oh!

gemini-code-assist Bot commented Dec 28, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	\| Llama \| meta-llama/Llama-4-Scout-17B-16E-Instruct,AI-ModelScope/Llama-3.1-8B-Instruct,LLM-Research/Llama-3.2-1B-Instruct \| <span style="color: red;">×</span> \| <span style="color: red;">×</span> \|
	\| Llama \| meta-llama/Llama-4-Scout-17B-16E-Instruct<br>AI-ModelScope/Llama-3.1-8B-Instruct<br>LLM-Research/Llama-3.2-1B-Instruct \| <span style="color: red;">×</span> \| <span style="color: red;">×</span> \|

	\| BGE-Reranker \| BAAI/bge-reranker-v2-m3 \| <span style="color: red;">×</span> \| <span style="color: red;">×</span> \|
	\| BGE-Reranker \| BAAI/bge-reranker-v2-m3 \| <span style="color: red;">×</span> \| <span style="color: red;">×</span> \|

Conversation

Hexq0210 commented Dec 28, 2025

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Uh oh!

gemini-code-assist Bot commented Dec 28, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants