[Model] Add Ernie4.5 VL model support by CSWYF3634076 · Pull Request #15679 · sgl-project/sglang

CSWYF3634076 · 2025-12-23T11:29:12Z

Motivation

Add Baidu Ernie4.5 VL model support

Modifications

ernie45_moe_vl.py the text backbone
ernie45_vl.py the vit
processors/ernie45_vl.py the processor
rotary_embedding.py::Ernie4_5_VLRotaryEmbedding the 3d_rope (hwhwhw...ttt..)

Accuracy Tests

python3 -m sglang.launch_server --model-path baidu/ERNIE-4.5-VL-28B-A3B-PT \
 --served-model-name ERNIE-45-VL-28B \ 
 --port 8301 \ 
 --trust-remote-code

ocrbench 88.60
countbench 87.37
docvqa 93.69
realworldqa 68.89

test case

curl --location --request POST 'http://127.0.0.1:8301/v1/chat/completions' \
--header 'Authorization: Bearer $OPENAI_API_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{
  "model": "ERNIE-45-VL-28B",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe the content of the image"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://paddlenlp.bj.bcebos.com/datasets/paddlemix/demo_images/example1.jpg"
          }
        }
      ]
    }
  ],
  "temperature": 1,
  "top_p": 1,
  "max_tokens": 1024,
  "skip_special_tokens": false,
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}'

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.
Work with maintainers to merge your PR. See the PR Merge Process

Others

I think the function process_mm_data should be rewritten by each Processor class, similar to the _call_hf_processor in vllm. I don't know why the current code doesn't do this

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

gemini-code-assist · 2025-12-23T11:29:34Z

Summary of Changes

Hello @CSWYF3634076, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for the Baidu Ernie4.5 VL model. It includes the necessary architectural components for both the text backbone (leveraging a Mixture-of-Experts approach) and the vision transformer, along with a dedicated multimodal processor. The changes enable the system to handle and process visual and textual inputs for the Ernie4.5 VL model, utilizing a specialized 3D rotary embedding for advanced multimodal positional encoding.

Highlights

Ernie4.5 VL Model Support: Added comprehensive support for the Baidu Ernie4.5 VL model, encompassing its text backbone, vision transformer, and multimodal processor.
New Chat Template: Introduced and registered a new chat template specifically designed for Ernie4.5 VL models, ensuring proper formatting of multimodal conversations.
3D Rotary Positional Embedding: Implemented a specialized 3D Rotary Positional Embedding (Ernie4_5_VLRotaryEmbedding) to effectively handle the complex positional encoding required for multimodal inputs (image/video and text).
Multimodal Architecture Integration: Integrated Ernie4_5_VLMoeForConditionalGeneration into the list of supported multimodal model architectures, allowing the system to recognize and utilize this new model.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds support for the Ernie4.5 VL model. The changes include new model definition files, a vision processor, and modifications to existing chat templates and model configurations. Overall, the implementation looks solid, but I've identified a few areas for improvement, including potential division-by-zero errors in the processor, inefficient tensor operations, and some code clarity issues. Addressing these points will enhance the robustness and performance of the new model integration.

gemini-code-assist · 2025-12-23T11:31:54Z

+    min_pixels: int = MIN_PIXELS,
+    max_pixels: int = MAX_PIXELS,
+):
+    if max(height, width) / min(height, width) > MAX_RATIO:


There is a potential ZeroDivisionError here if min(height, width) is 0. An image could theoretically have a height or width of 0. It's safer to add a check to prevent this.

min_dim = min(height, width) if min_dim == 0: return 0, 0 if max(height, width) / min_dim > MAX_RATIO:

gemini-code-assist · 2025-12-23T11:31:54Z

+        max_frames = floor_by_factor(
+            ele.get("max_frames", min(FPS_MAX_FRAMES, total_frames)), FRAME_FACTOR
+        )
+        nframes = total_frames / video_fps * fps


A ZeroDivisionError can occur here if video_fps is 0. It's important to add a check to handle this case to prevent the program from crashing.

if video_fps == 0: nframes = 0 else: nframes = total_frames / video_fps * fps

gemini-code-assist · 2025-12-23T11:31:54Z

+                device=input_ids.device,
+            )
+            image_index, video_index = 0, 0
+            for i, input_ids in enumerate(total_input_ids):


The loop variable input_ids shadows the function parameter with the same name defined on line 2196. This can be confusing and lead to bugs. It's a good practice to use a different name for the loop variable to improve code clarity.

Suggested change

for i, input_ids in enumerate(total_input_ids):

for i, current_input_ids in enumerate(total_input_ids):

gemini-code-assist · 2025-12-23T11:31:54Z

+                    input_type_group.append((key, start_index, end_index))
+
+                llm_pos_ids_list = []
+                video_frame_num = 1


The variable video_frame_num is initialized here and updated in the following loop (lines 2294, 2334, 2342), but it is never used. This appears to be dead code and should be removed to improve clarity.

gemini-code-assist · 2025-12-23T11:31:54Z

+        rope_scaling: Optional[Dict[str, Any]] = None,
+        rope_is_neox_style: bool = True,


The parameters rope_scaling and rope_is_neox_style are defined in the __init__ method's signature but are not used within the method. When creating Ernie4_5_VLRotaryEmbedding, is_neox_style is hardcoded to False and rope_scaling is not passed. This can be misleading and may cause issues if the model's configuration changes. Please either use these parameters or remove them if they are not needed.

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

yuan-luo · 2026-01-02T09:11:30Z

"I think the function process_mm_data should be rewritten by each Processor class, similar to the _call_cf_processor in vllm. I don't know why the current code doesn't do this"

I think it's because most VLMs are using process_mm_data_async which has been rewritten by each Processor class.

CSWYF3634076 · 2026-01-09T07:52:37Z

"I think the function process_mm_data should be rewritten by each Processor class, similar to the _call_cf_processor in vllm. I don't know why the current code doesn't do this"

I think it's because most VLMs are using process_mm_data_async which has been rewritten by each Processor class.

In the process_mm_data_async function, the process_and_combine_mm_data method ultimately invokes the process_mm_data function, which contains extensive if-else logic pertaining to the model name

CSWYF3634076 · 2026-01-09T07:53:42Z

@yuan-luo May I ask, besides resolving conflicts, is there anything else that needs to be updated in this PR

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

CSWYF3634076 · 2026-01-13T06:36:19Z

@yuan-luo Hi, could you please review this PR?

yuan-luo · 2026-01-14T03:34:47Z

@yuan-luo Hi, could you please review this PR?

Sure.

yuan-luo · 2026-01-14T03:42:53Z

"I think the function process_mm_data should be rewritten by each Processor class, similar to the _call_cf_processor in vllm. I don't know why the current code doesn't do this"
I think it's because most VLMs are using process_mm_data_async which has been rewritten by each Processor class.

In the process_mm_data_async function, the process_and_combine_mm_data method ultimately invokes the process_mm_data function, which contains extensive if-else logic pertaining to the model name

I guess you mean _call_hf_processor in vllm. Writing a _call_hf_processor in each model is a huge refactor which would introduce massive duplicated code segment. Probably the current mechanism is a concise and reusable implementation.

Signed-off-by: wangyafeng <wangyafeng@baidu.com>

CSWYF3634076 · 2026-01-16T07:41:37Z

@yuan-luo Hi, could you please review this PR?

Sure.

@yuan-luo
These are some implementation descriptions for easy review.

The ViT component is largely similar to the Qwen-VL ViT. The main difference is the use of self.resampler_model, which plays a role similar to self.merger in Qwen-VL. There are also some differences in weight naming.
In Ernie4_5_VLMoeForConditionalGeneration, the primary difference is that self.visual_token_mask is passed to the text model, allowing it to distinguish between experts during the forward pass.
In Ernie4_5_VLMoeModel, the main difference lies in the MoE stage, where routing decisions are made based on the token distribution (visual_token_mask) to select either text experts or visual experts. In addition, the Attention module differs by adopting 3D positional encoding (H, W, H, W, H, W …, T, T, T …).
In the processor component, the main difference is the reimplementation of process_mm_data, which includes additional post-processing steps to adapt to the fields required by the framework. In Ernie-VL, both images and videos share the same token_id, and MRotaryEmbedding.get_rope_index_ernie45 is used to generate positions.

yuan-luo · 2026-01-19T12:21:14Z

@yuan-luo Hi, could you please review this PR?

Sure.

@yuan-luo These are some implementation descriptions for easy review.

The ViT component is largely similar to the Qwen-VL ViT. The main difference is the use of self.resampler_model, which plays a role similar to self.merger in Qwen-VL. There are also some differences in weight naming.

In Ernie4_5_VLMoeForConditionalGeneration, the primary difference is that self.visual_token_mask is passed to the text model, allowing it to distinguish between experts during the forward pass.

In Ernie4_5_VLMoeModel, the main difference lies in the MoE stage, where routing decisions are made based on the token distribution (visual_token_mask) to select either text experts or visual experts. In addition, the Attention module differs by adopting 3D positional encoding (H, W, H, W, H, W …, T, T, T …).

In the processor component, the main difference is the reimplementation of process_mm_data, which includes additional post-processing steps to adapt to the fields required by the framework. In Ernie-VL, both images and videos share the same token_id, and MRotaryEmbedding.get_rope_index_ernie45 is used to generate positions.

I actually did exactly the same task on my own side, implementing this model, but encountered some error in correctness. So I can understand what obstacles you have pulled through, which is awesome. I'll review it ASAP.

yuan-luo

LGTM

CSWYF3634076 · 2026-01-21T11:24:24Z

@JustinTong0323 Hello, the review has been approved. Can you help trigger CI and subsequent merges

Signed-off-by: wangyafeng <wangyafeng@baidu.com>

JustinTong0323 · 2026-01-26T05:01:02Z

/tag-and-rerun-ci

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com> Signed-off-by: wangyafeng <wangyafeng@baidu.com>

CSWYF3634076 added 11 commits November 18, 2025 20:50

[Model] support ernie45 vl v1

f764289

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v2

24a5de8

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v3

7a387e2

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl vit v1

564241d

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl vit v2

b2c3308

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v3 text success

54c3c69

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v4 image success

c7371d9

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v5 video success

c823578

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v6 accuracy success

d63361a

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v7 code style

ca56c0a

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v8 code style

7ba4008

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

CSWYF3634076 requested review from BBuf, Edwardf0t1, Fridge003, HaiShaw, JustinTong0323, Ying1123, ch-wan, ispobock, merrymercy, mickqian and yhyang201 as code owners December 23, 2025 11:29

gemini-code-assist Bot reviewed Dec 23, 2025

View reviewed changes

[Model] support ernie45 vl v8 remove atten_backend

729d3cf

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

JustinTong0323 assigned yuan-luo Dec 29, 2025

yuan-luo self-requested a review January 2, 2026 09:11

CSWYF3634076 added 2 commits January 12, 2026 11:30

Merge branch 'main' into ernie-vl

49cba17

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v9 docs

ec66d29

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

github-actions Bot added documentation Improvements or additions to documentation Multi-modal multi-modal language model labels Jan 12, 2026

[Model] support ernie45 vl v10

8e95232

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com>

[Model] support ernie45 vl v11 remove unused code

dae50fb

Signed-off-by: wangyafeng <wangyafeng@baidu.com>

yuan-luo approved these changes Jan 21, 2026

View reviewed changes

yuan-luo approved these changes Jan 23, 2026

View reviewed changes

JustinTong0323 reviewed Jan 26, 2026

View reviewed changes

Comment thread python/sglang/lang/chat_template.py

[Model] support ernie45 vl v12 remove unused code chattemplate

504abca

Signed-off-by: wangyafeng <wangyafeng@baidu.com>

github-actions Bot added the run-ci label Jan 26, 2026

JustinTong0323 approved these changes Jan 26, 2026

View reviewed changes

Kangyan-Zhou merged commit 1a19b39 into sgl-project:main Jan 26, 2026
156 of 172 checks passed

Chen-0210 pushed a commit to Chen-0210/sglang that referenced this pull request Jan 30, 2026

[Model] Add Ernie4.5 VL model support (sgl-project#15679)

c71bdfe

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com> Signed-off-by: wangyafeng <wangyafeng@baidu.com>

Johnsonms pushed a commit to Johnsonms/sglang that referenced this pull request Feb 14, 2026

[Model] Add Ernie4.5 VL model support (sgl-project#15679)

d85fd85

Signed-off-by: CSWYF3634076 <wangyafeng@baidu.com> Signed-off-by: wangyafeng <wangyafeng@baidu.com>

	for i, input_ids in enumerate(total_input_ids):
	for i, current_input_ids in enumerate(total_input_ids):

		rope_scaling: Optional[Dict[str, Any]] = None,
		rope_is_neox_style: bool = True,

Conversation

CSWYF3634076 commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Others

Uh oh!

gemini-code-assist Bot commented Dec 23, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

yuan-luo commented Jan 2, 2026

Uh oh!

CSWYF3634076 commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CSWYF3634076 commented Jan 9, 2026

Uh oh!

CSWYF3634076 commented Jan 13, 2026

Uh oh!

yuan-luo commented Jan 14, 2026

Uh oh!

yuan-luo commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CSWYF3634076 commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuan-luo commented Jan 19, 2026

Uh oh!

yuan-luo left a comment

Choose a reason for hiding this comment

Uh oh!

CSWYF3634076 commented Jan 21, 2026

Uh oh!

Uh oh!

JustinTong0323 commented Jan 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CSWYF3634076 commented Dec 23, 2025 •

edited

Loading

CSWYF3634076 commented Jan 9, 2026 •

edited

Loading

yuan-luo commented Jan 14, 2026 •

edited

Loading

CSWYF3634076 commented Jan 16, 2026 •

edited

Loading