refactor: refactor engine vlm params#13069
Closed
minleminzui wants to merge 312 commits intosgl-project:refactor-engine-vlm-paramsfrom
Closed
refactor: refactor engine vlm params#13069minleminzui wants to merge 312 commits intosgl-project:refactor-engine-vlm-paramsfrom
minleminzui wants to merge 312 commits intosgl-project:refactor-engine-vlm-paramsfrom
Conversation
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
…ng (sgl-project#10702) Co-authored-by: a4zhangfei <a4zhangfei@qq.com>
…0218) (sgl-project#10225) Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
…(spec, non-spec, spec v2) x (retract, finished)` (sgl-project#12224)
Co-authored-by: Hubert Lu <Hubert.Lu@amd.com>
…r 8-gpu-h200 runners (sgl-project#12952)
Co-authored-by: HAI <hixiao@gmail.com> Co-authored-by: Hubert Lu <55214931+hubertlu-tw@users.noreply.github.com>
…ests
- MRotaryEmbedding:
- Make audio-related kwargs (audio_token_id, audio_start_token_id, position_id_per_seconds, audio_seqlens) optional.
- Introduce `has_audio` guard and skip audio-specific branches when no audio is present.
- Ensure all newly created position_id tensors (`torch.arange`) are allocated on the same device as input_ids.
- mm_utils:
- Change `_adjust_embedding_length` to take `special_multimodal_mask` instead of a generic mask.
- Handle length mismatch robustly: pad with zeros when embeddings are shorter than the number of multimodal tokens, truncate when longer.
- Replace hard RuntimeError with warnings and best-effort adjustment to avoid crashing on imperfect MM embeddings.
- Qwen3VL:
- Rework `get_image_feature` to explicitly reconstruct patches based on `in_channels * temporal_patch_size * patch_size^2`.
- Enforce full patch and patch-group (spatial_merge_size^2) alignment, skipping invalid/too-short inputs.
- Return empty tensor when there are no valid image patches, and ensure all tensors are on the visual module’s device/dtype.
- Tests (OpenAI vision server):
- Override `verify_single_image_response` in `TestQwen3OmniServer` to only check high-level structure:
- presence of “1.” and “2.”,
- mentions of image/picture/photo and audio/sound/speech,
- valid usage stats.
- Add a Qwen3-Omni-specific `verify_speech_recognition_response` that checks structural/audio mentions instead of exact transcript words.
- Fix a bug in common `verify_single_image_response` where `"person"` was not actually checked with `in text`.
…ature - Relaxed shape check in `get_image_feature`: allow `pixel_values` with dim > 2 (e.g. `[B, T, D]` or `[B, H, W, C]`) instead of hard-asserting `dim() == 2` - Flatten all leading dims into a single batch dim to match `[N, D]` expected by `self.visual` - Keeps backward compatibility for existing `[N, D]` image embeddings - Fixes AssertionError(3) raised when running Qwen3-Omni mixed-modality tests - Verified passing `TestQwen3OmniServer::test_mixed_modality_chat_completion`
…rver tests In TestOpenAIMLLMServerBase.setUpClass, raise unittest.SkipTest when model is missing/empty Prevents pytest from collecting/execing mixin classes and throwing AttributeError: ... has no attribute 'model' Keeps CI green by marking mixins as skipped instead of erroring No impact on concrete test classes that define model
a0563ed to
0e09b6a
Compare
…st_mixed_modality_chat_completion
…ulti_images_chat_completion
2904ebf to
20ebb81
Compare
…ideo_images_chat_completion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#10532
Mainly to ensure CI stability for unit-test-backend-1-gpu (0),
which runs
pytest test/srt/test_vision_openai_server_a.py.Fix Qwen3-Omni multimodal embedding handling and relax vision/audio tests
MRotaryEmbedding:
has_audioguard and skip audio-specific branches when no audio is present.torch.arange) are allocated on the same device as input_ids.mm_utils:
_adjust_embedding_lengthto takespecial_multimodal_maskinstead of a generic mask.Qwen3VL:
get_image_featureto explicitly reconstruct patches based onin_channels * temporal_patch_size * patch_size^2.Tests (OpenAI vision server):
verify_single_image_responseinTestQwen3OmniServerto only check high-level structure:verify_speech_recognition_responsethat checks structural/audio mentions instead of exact transcript words.verify_single_image_responsewhere"person"was not actually checked within text.Modifications
Accuracy Tests
Benchmarking and Profiling
Checklist