Questions

1. From my understanding of current implementation of sglang, it does not support two kind of attentions at the same time, thus the attention module in Resampler can't be replaced with RadixAttention. Is that correct?

Questions

1. From my understanding on current implementation of sglang, it does not support two kind of attentions at the same time, thus the attention module in Resampler and Idefics2VisionAttention can't be replaced with RadixAttention. Is that correct?
2. Current implementation of Idefics2VisionAttention in this PR relies on vllm.attention.MultiheadAttention, which is introduced in version vllm==0.6.5. If that is not acceptable, I will rewrite the attention module

@Ying1123 @merrymercy

@mickqian Looks nice! Are there anything left to do?

@mickqian Looks nice! Are there anything left to do?看起来不错！还有什么要做的吗？

1. The number of frames processed from the video is hard-coded for the moment, which should be calculated. I added a TODO on it.
2. A CI Test failed because of this:

[2025-01-10 03:24:51 TP0] Multimodal prompt is too long after expanding multimodal tokens. After expanding len(req.origin_input_ids_unpadded)=29 => 308 >= 294.

from an assertion of checking the input_ids' length. The token limit (from my understanding) is calculated from the GPU:

 if len(req.origin_input_ids) >= self.max_req_input_len:
        logger.error(
            "Multimodal prompt is too long after expanding multimodal tokens. "
            f"After expanding {len(req.origin_input_ids_unpadded)=} => {len(req.origin_input_ids)} >= {self.max_req_input_len}. "
                )

3. Some mp4 file failed the decode process in decode_video_base64

@hnyls2002 Could you help to review. Thanks!

@mickqian Thanks. We will update the API key as soon as possible. Before that, could you fix the conflicts of qwen2_vl.py.

LGTM cc @merrymercy

@mickqian Also, remove Chinese comments in the PR. Thanks so much. Could we try not import any vllm dependency, rather rewrite it ourselves.

Under the models files, we prefer not to import anything from vllm. We will remove them all later.

Trying to address these in #2977