Skip to content

refactor: bug fixes and refactor for vlm#4661

Merged
zhyncs merged 2 commits intosgl-project:mainfrom
mickqian:vlm-refactor
Mar 23, 2025
Merged

refactor: bug fixes and refactor for vlm#4661
zhyncs merged 2 commits intosgl-project:mainfrom
mickqian:vlm-refactor

Conversation

@mickqian
Copy link
Copy Markdown
Collaborator

@mickqian mickqian commented Mar 22, 2025 •
edited
Loading

Motivation

  1. a general vlm embed routine
  2. sglang image processors architecture improvement
  3. fix the bug of registering customized image processor to transformers
  4. add some consistency to vlm api

Modifications

Checklist

mickqian
mickqian commented Mar 22, 2025
View reviewed changes
Comment thread python/sglang/srt/managers/mm_utils.py Outdated
Copy link
Copy Markdown
Collaborator Author

@mickqian mickqian Mar 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this modification, the result of mmmu benchmark of sglang goes from:

accuracy time elapsed (s)
Qwen2.5-VL-7B-Instruct 0.462 229.1
gemma-3-4b-it 0.409 226.4

to:

accuracy time elapsed (s)
Qwen2.5-VL-7B-Instruct 0.477 230.2
gemma-3-4b-it 0.409 202.3

@mickqian
Copy link
Copy Markdown
Collaborator Author

mickqian commented Mar 22, 2025 •
edited
Loading

This should fix #4607 #4159 and many other similar issues I can't count

@mickqian mickqian mentioned this pull request Mar 22, 2025
Merged
4 tasks
@zhaochenyang20
Copy link
Copy Markdown
Collaborator

zhaochenyang20 commented Mar 22, 2025

@mickqian I've rerun the CI. thanks! minicpm-o is of high-priority qvq

@mickqian mickqian changed the title refactor: bug fixes and refactor of vlm refactor: bug fixes and refactor for vlm Mar 22, 2025
@mickqian mickqian force-pushed the vlm-refactor branch 6 times, most recently from d64a33b to f644731 Compare March 22, 2025 14:45
@zhaochenyang20
Copy link
Copy Markdown
Collaborator

zhaochenyang20 commented Mar 22, 2025

@mickqian this is not ready to review right? mick, if ready, please ping yi and me

@mickqian mickqian force-pushed the vlm-refactor branch 4 times, most recently from 61f6c15 to 0594b06 Compare March 22, 2025 16:49
@mickqian
Copy link
Copy Markdown
Collaborator Author

mickqian commented Mar 22, 2025

@yizhang2077 @zhaochenyang20 ready now. vlm tests all passed. Thanks

@mickqian mickqian force-pushed the vlm-refactor branch 2 times, most recently from e12c107 to ae6735f Compare March 22, 2025 17:16
yizhang2077
yizhang2077 reviewed Mar 22, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@yizhang2077 yizhang2077 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add some basic comments here, continue to reviewing

Comment thread test/srt/run_suite.py Outdated
Comment thread python/sglang/srt/configs/utils.py Outdated
Comment thread python/sglang/srt/models/gemma3_mm.py Outdated
Comment thread python/sglang/srt/models/minicpmv.py Outdated
Comment thread python/sglang/srt/models/qwen2_5_vl.py Outdated
mickqian
mickqian commented Mar 23, 2025
View reviewed changes
Comment thread python/sglang/srt/conversation.py Outdated
yizhang2077
yizhang2077 reviewed Mar 23, 2025
View reviewed changes
Comment thread test/srt/test_vlm_accuracy.py Outdated
Comment thread python/sglang/srt/managers/schedule_batch.py Outdated
@yizhang2077
Copy link
Copy Markdown
Collaborator

yizhang2077 commented Mar 23, 2025 •
edited
Loading

LGTM, great code refactoring! After CI pass, this PR can be merged cc @zhaochenyang20

@yizhang2077 yizhang2077 mentioned this pull request Mar 23, 2025
Closed
6 tasks
@zhaochenyang20
Copy link
Copy Markdown
Collaborator

zhaochenyang20 commented Mar 23, 2025

@mickqian @yizhang2077 great!

@zhyncs zhyncs merged commit 11577ce into sgl-project:main Mar 23, 2025
@vhain vhain mentioned this pull request Mar 27, 2025
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yizhang2077 yizhang2077 yizhang2077 approved these changes

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu

@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mickqian @zhaochenyang20 @yizhang2077 @zhyncs