model: Support Janus-pro by mickqian · Pull Request #3203 · sgl-project/sglang

mickqian · 2025-01-29T04:28:55Z

Motivation

Support deepseek-ai/Janus-Pro models, #3195

Modifications

Janus model files
related tests

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling.

zhyncs · 2025-01-29T07:14:00Z

Coooooool!

FocusAware · 2025-01-31T13:16:04Z

Forgive my noobishness, but how to test the PR?
I run:

python -m sglang.launch_server --model-path deepseek-ai/Janus-Pro-7B --trust-remote-code --chat-template
 janus --port 30000 --host 0.0.0.0

And it aborts with the error below:

  File "/home/user/tools/sglang/python/sglang/srt/models/registry.py", line 65, in resolve_model_cls
    return self._raise_for_unsupported(architectures)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tools/sglang/python/sglang/srt/models/registry.py", line 32, in _raise_for_unsupported
    raise ValueError(ValueError: Model architectures ['MultiModalityCausalLM'] are not supported for now. Supported architectures: dict_keys(['BaichuanForCausalLM', 'ChatGLMModel', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'DbrxForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'DeepseekV3ForCausalLM', 'ExaoneForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'Gemma2ForSequenceClassification', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GraniteForCausalLM', 'Grok1ForCausalLM', 'Grok1ModelForCausalLM', 'InternLM2ForCausalLM', 'InternLM2ForRewardModel', 'LlamaForCausalLM', 'Phi3ForCausalLM', 'InternLM3ForCausalLM', 'LlamaForClassification', 'LlamaForCausalLMEagle', 'LlamaEmbeddingModel', 'MistralModel', 'LlamaForSequenceClassification', 'LlamaForSequenceClassificationWithNormal_Weights', 'LlavaLlamaForCausalLM', 'LlavaQwenForCausalLM', 'LlavaMistralForCausalLM', 'LlavaVidForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'MiniCPMV', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MllamaForConditionalGeneration', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'OlmoeForCausalLM', 'Phi3SmallForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2ForCausalLMEagle', 'Qwen2MoeForCausalLM', 'Qwen2VLForConditionalGeneration', 'StableLmForCausalLM', 'TorchNativeLlamaForCausalLM', 'TorchNativePhi3ForCausalLM', 'XverseForCausalLM', 'XverseMoeForCausalLM', 'YiVLForCausalLM'])

[2025-01-31 13:12:23] Received sigquit from a child proces. It usually means the child failed.
Killed

mickqian · 2025-02-01T06:58:32Z

Forgive my noobishness, but how to test the PR? I run:

python -m sglang.launch_server --model-path deepseek-ai/Janus-Pro-7B --trust-remote-code --chat-template
 janus --port 30000 --host 0.0.0.0

And it aborts with the error below:

  File "/home/user/tools/sglang/python/sglang/srt/models/registry.py", line 65, in resolve_model_cls
    return self._raise_for_unsupported(architectures)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tools/sglang/python/sglang/srt/models/registry.py", line 32, in _raise_for_unsupported
    raise ValueError(ValueError: Model architectures ['MultiModalityCausalLM'] are not supported for now. Supported architectures: dict_keys(['BaichuanForCausalLM', 'ChatGLMModel', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'DbrxForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'DeepseekV3ForCausalLM', 'ExaoneForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'Gemma2ForSequenceClassification', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GraniteForCausalLM', 'Grok1ForCausalLM', 'Grok1ModelForCausalLM', 'InternLM2ForCausalLM', 'InternLM2ForRewardModel', 'LlamaForCausalLM', 'Phi3ForCausalLM', 'InternLM3ForCausalLM', 'LlamaForClassification', 'LlamaForCausalLMEagle', 'LlamaEmbeddingModel', 'MistralModel', 'LlamaForSequenceClassification', 'LlamaForSequenceClassificationWithNormal_Weights', 'LlavaLlamaForCausalLM', 'LlavaQwenForCausalLM', 'LlavaMistralForCausalLM', 'LlavaVidForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'MiniCPMV', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MllamaForConditionalGeneration', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'OlmoeForCausalLM', 'Phi3SmallForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2ForCausalLMEagle', 'Qwen2MoeForCausalLM', 'Qwen2VLForConditionalGeneration', 'StableLmForCausalLM', 'TorchNativeLlamaForCausalLM', 'TorchNativePhi3ForCausalLM', 'XverseForCausalLM', 'XverseMoeForCausalLM', 'YiVLForCausalLM'])

[2025-01-31 13:12:23] Received sigquit from a child proces. It usually means the child failed.
Killed

Could you make sure the local repo has been updated, and python has chosen the correct sglang version? BTW, this PR will be unstable until merged.

FocusAware · 2025-02-01T16:03:56Z

Forgive my noobishness, but how to test the PR? I run:

python -m sglang.launch_server --model-path deepseek-ai/Janus-Pro-7B --trust-remote-code --chat-template
 janus --port 30000 --host 0.0.0.0

And it aborts with the error below:

  File "/home/user/tools/sglang/python/sglang/srt/models/registry.py", line 65, in resolve_model_cls
    return self._raise_for_unsupported(architectures)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tools/sglang/python/sglang/srt/models/registry.py", line 32, in _raise_for_unsupported
    raise ValueError(ValueError: Model architectures ['MultiModalityCausalLM'] are not supported for now. Supported architectures: dict_keys(['BaichuanForCausalLM', 'ChatGLMModel', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'DbrxForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'DeepseekV3ForCausalLM', 'ExaoneForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'Gemma2ForSequenceClassification', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GraniteForCausalLM', 'Grok1ForCausalLM', 'Grok1ModelForCausalLM', 'InternLM2ForCausalLM', 'InternLM2ForRewardModel', 'LlamaForCausalLM', 'Phi3ForCausalLM', 'InternLM3ForCausalLM', 'LlamaForClassification', 'LlamaForCausalLMEagle', 'LlamaEmbeddingModel', 'MistralModel', 'LlamaForSequenceClassification', 'LlamaForSequenceClassificationWithNormal_Weights', 'LlavaLlamaForCausalLM', 'LlavaQwenForCausalLM', 'LlavaMistralForCausalLM', 'LlavaVidForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'MiniCPMV', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MllamaForConditionalGeneration', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'OlmoeForCausalLM', 'Phi3SmallForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2ForCausalLMEagle', 'Qwen2MoeForCausalLM', 'Qwen2VLForConditionalGeneration', 'StableLmForCausalLM', 'TorchNativeLlamaForCausalLM', 'TorchNativePhi3ForCausalLM', 'XverseForCausalLM', 'XverseMoeForCausalLM', 'YiVLForCausalLM'])

[2025-01-31 13:12:23] Received sigquit from a child proces. It usually means the child failed.
Killed

Could you make sure the local repo has been updated, and python has chosen the correct sglang version? BTW, this PR will be unstable until merged.

The issue was caused by a missing dependency. Managed to get it working after running "pip install addict".

FocusAware · 2025-02-01T16:26:08Z

Any plans to add text to image?

yizhang2077

Sorry for late review, I take a basic review through it and leave some comments here, I'll take a closer look these days.

zhaochenyang20 · 2025-03-09T04:02:49Z

@mickqian Mick. Could you decouple this PR into two or more?

At least one for refactor, second for the model?

zhaochenyang20 · 2025-03-12T02:58:35Z

@mickqian rebase with the main?

zhaochenyang20 · 2025-03-12T04:21:51Z

@mickqian @yizhang2077 why after refactor the codes and merged a small PR, this PR is still huge? 3000+ lines

mickqian · 2025-03-12T04:35:41Z

@zhyncs All the model class/config class/processor classes takes up to 2000+ lines. The model class alone is 2000+ lines

zhaochenyang20 · 2025-03-12T04:45:41Z

@yizhang2077 if you approve and pass the CI, I can merge it

yizhang2077

Why we update deepgeem here?

mickqian · 2025-03-12T06:47:26Z

Why we update deepgeem here?

updated

zhaochenyang20 · 2025-03-13T09:06:14Z

congrats

mickqian force-pushed the janus-pro branch from 250e105 to 0183d9d Compare January 29, 2025 04:38

mickqian marked this pull request as ready for review January 29, 2025 07:13

mickqian requested review from ByronHsu, Ying1123, hnyls2002, ispobock, merrymercy and zhyncs as code owners January 29, 2025 07:13

zhyncs assigned yizhang2077 Jan 29, 2025

mickqian force-pushed the janus-pro branch 8 times, most recently from b6542eb to a29f545 Compare January 31, 2025 13:10

mickqian force-pushed the janus-pro branch from a29f545 to 8d47848 Compare February 2, 2025 13:58

mickqian mentioned this pull request Feb 6, 2025

model: Intern vl 2.5 #3351

Closed

5 tasks

yizhang2077 reviewed Feb 12, 2025

View reviewed changes

Comment thread python/sglang/test/test_utils.py Outdated

Comment thread test/srt/test_vision_openai_server.py Outdated

Comment thread python/sglang/srt/hf_transformers_utils.py Outdated

Comment thread python/sglang/srt/hf_transformers_utils.py Outdated

yizhang2077 reviewed Feb 12, 2025

View reviewed changes

Comment thread python/sglang/srt/managers/schedule_batch.py Outdated

mickqian force-pushed the janus-pro branch from 8d47848 to ae142ec Compare February 13, 2025 03:19

mickqian requested review from BBuf and HandH1998 as code owners February 13, 2025 03:19

mickqian force-pushed the janus-pro branch from ae142ec to 34028ad Compare February 13, 2025 03:20

zhaochenyang20 mentioned this pull request Mar 4, 2025

Development Roadmap (2025 H1) #4042

Closed

67 tasks

mickqian force-pushed the janus-pro branch 2 times, most recently from 8e6765d to 931c250 Compare March 4, 2025 02:20

merrymercy requested a review from zhaochenyang20 as a code owner March 8, 2025 06:12

mickqian force-pushed the janus-pro branch 3 times, most recently from 0c5771e to ae2529c Compare March 12, 2025 01:26

mickqian force-pushed the janus-pro branch from ae2529c to 83517e5 Compare March 12, 2025 03:10

mickqian force-pushed the janus-pro branch 2 times, most recently from 104e497 to 6336345 Compare March 12, 2025 04:35

yizhang2077 reviewed Mar 12, 2025

View reviewed changes

mickqian force-pushed the janus-pro branch from 16bdde0 to 5f154fa Compare March 12, 2025 06:47

model: Support Janus-pro

32a1cc0

mickqian force-pushed the janus-pro branch from 066bb7c to 32a1cc0 Compare March 12, 2025 07:31

yizhang2077 added 2 commits March 12, 2025 17:24

Merge branch 'main' into janus-pro

684e429

Merge branch 'main' into janus-pro

0ae26d2

yizhang2077 approved these changes Mar 12, 2025

View reviewed changes

mickqian requested a review from yizhang2077 March 12, 2025 15:36

zhaochenyang20 approved these changes Mar 12, 2025

View reviewed changes

zhyncs merged commit 01090e8 into sgl-project:main Mar 12, 2025

hebiao064 pushed a commit to hebiao064/sglang that referenced this pull request Mar 13, 2025

model: Support Janus-pro (sgl-project#3203)

9ff048a

m0g1cian mentioned this pull request Mar 18, 2025

[Bug] Cannot run Janus Pro #4534

Closed

5 tasks

vhain mentioned this pull request Mar 27, 2025

deps: lazy import optional dependencies gguf and torchvision #4826

Merged

6 tasks

Conversation

mickqian commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Checklist

Uh oh!

zhyncs commented Jan 29, 2025

Uh oh!

FocusAware commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mickqian commented Feb 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FocusAware commented Feb 1, 2025

Uh oh!

FocusAware commented Feb 1, 2025

Uh oh!

yizhang2077 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhaochenyang20 commented Mar 9, 2025

Uh oh!

zhaochenyang20 commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhaochenyang20 commented Mar 12, 2025

Uh oh!

mickqian commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhaochenyang20 commented Mar 12, 2025

Uh oh!

yizhang2077 left a comment

Choose a reason for hiding this comment

Uh oh!

mickqian commented Mar 12, 2025

Uh oh!

zhaochenyang20 commented Mar 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mickqian commented Jan 29, 2025 •

edited

Loading

FocusAware commented Jan 31, 2025 •

edited

Loading

mickqian commented Feb 1, 2025 •

edited

Loading

yizhang2077 left a comment •

edited

Loading

zhaochenyang20 commented Mar 12, 2025 •

edited

Loading

mickqian commented Mar 12, 2025 •

edited

Loading