Motivation

• Support encoder-decoder architecture in SGLang.
• Support llama vision model.
• Support CUDA graph and prefix cache for llama vision model

Note that to support CUDA graph for encoder-decoder architecture like llama vision (mllama), we should make encoder_lens the part of the cuda graph, as the full_text_row_masked_out_mask is decided by encoder_lens to skip the text-only req in a mixed batch.

However, the current cuda graph backend (flashinfer) seems to have trouble handling the mixed batch. So we for now only accept the pure image decoding batch.

Todo in the following PRs:

• Split attention backends: sliding_window, single_attention, cross_attention
• Optimize encoder cache locations indexing, and reduce memory usage.

Modifications

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.