[Bugfix] Fix Qwen3-VL timestamp mismatch when using num_frames without fps by weiguangli-io · Pull Request #36136 · vllm-project/vllm

weiguangli-io · 2026-03-05T12:31:15Z

Summary

When num_frames is provided via mm_processor_kwargs without fps, _get_video_second_idx was falling back to the default fps (2) to compute the number of frames for timestamp calculation. However, HF's Qwen3VLVideoProcessor.sample_frames treats num_frames and fps as mutually exclusive — when num_frames is provided, it uses it directly. This caused a mismatch between the computed timestamps length and the actual video_grid_thw frame count, triggering:

AssertionError: The timestamps length(10) should be equal video length (16).

Changes:

Add sampled_num_frames parameter to _get_video_second_idx in Qwen3VLProcessingInfo
When num_frames is explicitly provided, use it directly instead of computing from fps, mirroring HF's behavior
Pass num_frames from mm_processor_kwargs through to _get_video_second_idx in _call_hf_processor

This fix also applies to Qwen3.5-VL which inherits Qwen3VLProcessingInfo.

Test plan

Verify with the reproduction script from the issue: pass mm_processor_kwargs={"num_frames": 32, "do_sample_frames": True} to llm.chat() with a video input on Qwen3-VL
Verify that existing fps-based sampling still works correctly (no num_frames provided)
Verify that passing both num_frames and fps raises an error in HF processor (existing behavior, unchanged)

gemini-code-assist

Code Review

This pull request addresses a bug in Qwen3-VL and Qwen3.5-VL related to a timestamp mismatch when num_frames is specified without fps. The changes ensure that if num_frames is provided, it is used directly for timestamp calculations, which aligns with the behavior of the underlying Hugging Face processor and resolves the AssertionError. The implementation is clean and directly fixes the issue by passing num_frames through to the timestamp generation logic. The changes look correct and I have no further feedback.

Isotr0py

Thanks, can you add a processor test under tests/models/multimodal/processing to avoid regression?

DarkLight1337 · 2026-03-06T06:16:45Z

PTAL at the failing test: https://buildkite.com/vllm/ci/builds/54843/steps/canvas?jid=019cc179-6f6a-4050-9115-2b2406f3e3e4#019cc179-6f6a-4050-9115-2b2406f3e3e4

weiguangli-io · 2026-03-08T03:13:37Z

Hi @DarkLight1337, I checked the failing CI — the multi-modal-processor-test-cpu failure appears to be related to my changes. I'll investigate the Buildkite log and push a fix. The AMD failures look like unrelated flakes. Thanks for the heads up!

mergify · 2026-03-08T03:36:54Z

Hi @OiPunk, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

weiguangli-io · 2026-03-08T12:34:29Z

CI Failure Analysis

Failing check: language-models-tests-extra-standard-1 (Buildkite build #55133)

Verdict: Unrelated flaky test — no code changes needed.

Evidence:

Not related to PR changes — This test runs pytest -v -s models/language -m 'core_model and slow_test', testing language models. This PR only modifies vllm/model_executor/models/qwen3_vl.py (a multimodal model) and adds a test in tests/models/multimodal/processing/test_qwen3_vl.py. The test is triggered because source_file_dependencies includes vllm/model_executor/models/, but the actual test content has no relation to Qwen3-VL.
Known flaky test — Issue [CI Failure]: mi325_8: Language Models Tests (Extra Standard) %N #29458 documents previous CI failures for the same Language Models Tests (Extra Standard) %N suite.
Main branch also failing — The latest nightly full CI build on main (build #55120) also has 2 failures, confirming pre-existing instability.
Relevant tests pass — The multi-modal-processor (GPU) test, which includes processor tests for Qwen3-VL, has already passed. The multi-modal-processor-test-cpu is still running.

weiguangli-io · 2026-03-09T13:09:54Z

@DarkLight1337 I've investigated the failing CI. The language-models-tests-extra-standard-1 failure is unrelated to my changes (detailed analysis in my earlier comment). The relevant multi-modal-processor tests have all passed. Could you please merge this or trigger a re-run? Thank you!

…t fps When `num_frames` is provided via `mm_processor_kwargs` without `fps`, `_get_video_second_idx` was falling back to the default fps (2) to compute the number of frames for timestamp calculation. However, the HF processor uses `num_frames` directly (since `num_frames` and `fps` are mutually exclusive in `Qwen3VLVideoProcessor.sample_frames`). This caused a mismatch between the computed timestamps length and the actual `video_grid_thw` frame count, triggering an assertion error. Pass `num_frames` through to `_get_video_second_idx` and use it directly when provided, mirroring the HF processor's behavior. Also explicitly set `fps=None` when `num_frames` is specified to prevent HF's `BaseVideoProcessor.preprocess()` from filling in the class default (`fps=2`) via `setdefault()`. Also adds processor regression tests for Qwen3-VL num_frames timestamp handling, with video metadata aligned to ndarray bounds. Fixes vllm-project#35909 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: OiPunk <codingpunk@gmail.com>

…t fps (vllm-project#36136) Signed-off-by: OiPunk <codingpunk@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…t fps (vllm-project#36136) Signed-off-by: OiPunk <codingpunk@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>

weiguangli-io requested a review from sighingnow as a code owner March 5, 2026 12:31

mergify Bot added qwen Related to Qwen models bug Something isn't working labels Mar 5, 2026

gemini-code-assist Bot reviewed Mar 5, 2026

View reviewed changes

weiguangli-io force-pushed the codex/vllm-35909-fix-num-frames-timestamps branch from 5add9ee to 896d89b Compare March 5, 2026 13:23

DarkLight1337 requested a review from Isotr0py March 5, 2026 16:40

Isotr0py reviewed Mar 5, 2026

View reviewed changes

weiguangli-io requested review from DarkLight1337 and ywang96 as code owners March 6, 2026 02:32

mergify Bot added the multi-modality Related to multi-modality (#4194) label Mar 6, 2026

Isotr0py approved these changes Mar 6, 2026

View reviewed changes

Isotr0py enabled auto-merge (squash) March 6, 2026 04:33

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 6, 2026

auto-merge was automatically disabled March 6, 2026 09:59
Head branch was pushed to by a user without write access

weiguangli-io force-pushed the codex/vllm-35909-fix-num-frames-timestamps branch 3 times, most recently from 58b6d2f to 14d7a3d Compare March 7, 2026 04:11

weiguangli-io force-pushed the codex/vllm-35909-fix-num-frames-timestamps branch from 00b007f to b7ba987 Compare March 8, 2026 11:43

weiguangli-io force-pushed the codex/vllm-35909-fix-num-frames-timestamps branch from b7ba987 to df3f36d Compare March 10, 2026 12:49

vllm-bot merged commit 7247596 into vllm-project:main Mar 11, 2026
59 of 63 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Bugfix] Fix Qwen3-VL timestamp mismatch when using num_frames without fps#36136

[Bugfix] Fix Qwen3-VL timestamp mismatch when using num_frames without fps#36136
vllm-bot merged 1 commit into
vllm-project:mainfrom
weiguangli-io:codex/vllm-35909-fix-num-frames-timestamps

weiguangli-io commented Mar 5, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Isotr0py left a comment

Uh oh!

DarkLight1337 commented Mar 6, 2026

Uh oh!

weiguangli-io commented Mar 8, 2026

Uh oh!

mergify Bot commented Mar 8, 2026

Uh oh!

weiguangli-io commented Mar 8, 2026

Uh oh!

weiguangli-io commented Mar 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

weiguangli-io commented Mar 5, 2026

Summary

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented Mar 6, 2026

Uh oh!

weiguangli-io commented Mar 8, 2026

Uh oh!

mergify Bot commented Mar 8, 2026

Uh oh!

weiguangli-io commented Mar 8, 2026

CI Failure Analysis

Uh oh!

weiguangli-io commented Mar 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants