👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

Documentation preview: https://vllm--30550.org.readthedocs.build/en/30550/

Hi @jzakrzew, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

hello @Samoed

How does MTEB handle score templates? We are looking to align our implementation with MTEB.

More links related to score templates:

• [New Model]: Support Qwen3 Embedding & Reranker  #19260
• [Model][Last/4] Automatic conversion of CrossEncoding model #19675
• [Bug]: Qwen3-Reranker issue with score #21681 (comment)
• [Frontend][Model] Qwen3Rerank API Server backward compatibility #20239
• [Feat] CrossEncoder class accept prompt template parameters huggingface/sentence-transformers#3406
• [Model][VLM] Support JinaVL Reranker #20260

Let's find a way to resolve it once for all.

Hi! We have a separate class for handling instruction-based models that process instructions, with an example for Qwen3. However, this approach is a bit naive, since there's no standard way of doing this yet. Maybe @tomaarsen has some thoughts on standardizing prompt templates for cross-encoders

For me, always unclear why there are no models that defines prompts in some jinja templates that could be used more automatically

hello @tomaarsen

Please take a look at this thread.

Thanks for pinging me @noooop & @Samoed.
In Sentence Transformers, I've been wanting to support the modern Causal-style rerankers, specifically:

• https://huggingface.co/BAAI/bge-reranker-v2.5-gemma2-lightweight
• https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2
• https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2
• https://huggingface.co/Qwen/Qwen3-Reranker-0.6B
• https://huggingface.co/Qwen/Qwen3-Reranker-4B
• https://huggingface.co/Qwen/Qwen3-Reranker-8B
• https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-1b
• https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b
• https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-6b
• https://huggingface.co/nvidia/llama-nemotron-rerank-1b-v2

As this modern format is becoming a lot more prevalent. For my codebase, there were always two main concerns:

1. The CrossEncoder (a.k.a. reranker) class relies on the transformers AutoModelForSequenceClassification. Models loaded with this factory are actually rather similar to AutoModelForCausalLM models, but their classifier head predicts one class akin to regression, rather than predicting scores for all tokens in the vocabulary and 1) taking the score for yes or 1 or 2) taking the difference of the scores of yes and no or 1 and 0 (which is also why you can make https://huggingface.co/tomaarsen/Qwen3-Reranker-0.6B-seq-cls out of https://huggingface.co/Qwen/Qwen3-Reranker-0.6B).
2. These models rely on a template, rather than the traditional text pairs input that was possible for AutoModelForSequenceClassification.

I think working on this support is so important that I'm working on a major refactor of the Sentence Transformers codebase, notably around the CrossEncoder, to help modularize it. This allows me to very easily support models that don't rely on AutoModelForSequenceClassification via built-in or custom modules that are executed sequentially. That solves concern 1 that I had, which is rather unrelated to this vLLM issue, but might give some more context to my recent work on the rerankers.

For concern 2, a very simple solution is to rely on the transformers chat_template, where the two texts passed to rerank are passed as two "messages" in a single text-chain. The format could then look something akin to:

<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: Given a web search query, retrieve relevant passages that answer the query
<Query>: {{ messages[0]["content"] }}
<Document>: {{ messages[1]["content"] }}<|im_end|>
<|im_start|>assistant
<think>\n\n</think>\n\n\n

[
    {
        "role": "query",
        "content": "What is the capital of France?",
    },
    {
        "role": "document",
        "content": "Paris is the capital of France.",
    }
]
tokenized = tokenizer.apply_chat_template(messages, ...)

(This one matches the format required for https://huggingface.co/Qwen/Qwen3-Reranker-0.6B, I believe)

An additional benefit here is that we can take advantage of a "system prompt" of sorts as an instruction/prompt for the reranker. In the above chat template, I hardcoded Given a web search query, retrieve relevant passages that answer the query, but that is the instruction that users could modify to match their use cases.

Some of the obvious advantages is that the transformers chat_template is pretty commonly used and known, that most LLM-based models would already have a chat-template copied from the base model that can very reasonably be updated (the chat-template for generation is useless on a reranker after all), etc.

But, my primary hesitation at the current stage is that apply_chat_template doesn't allow for any template-aware truncation. I suppose this is more often an issue on the LLM side, but the current template-unaware truncation will strip off crucial template tokens (e.g. <think>\n\n</think>\n\n\n), at which point the reranker scores become absolutely useless. This wouldn't be much of an issue if reranker models didn't impose any sequence length limits, but most do (e.g. 32k for Qwen3, 1024 for BAAI, 8k/32k for Mixedbread, etc.).

This tempts me to write a more "manual" templating implementation, where I can apply the truncation on the second input (often the 'document' in a query-document setting). Recurring issues that I've found with my initial attempts are that you can't fully separately tokenize the template from the actual texts, as many template tokens will want to "merge" with actual text tokens (e.g. having : Hello as a token in ... Query: {{ messages[0]["query"] }} ...)

Those are my thoughts for now. @noooop , where do you stand regarding:

1. using transformers' chat_template as the source of truth for the templating?
2. whether truncation limits are a problem, or whether you think we can ignore it? Your experience with LLMs that more frequently rely on chat templates might be useful there.

• Tom Aarsen

using transformers' chat_template as the source of truth for the templating?

Generally I think this should be like it, but for now there are now models. Even Qwen just inherit template from original LLM.

"role": "document",

Generally I think good approach, but I'm afraid some libraries won't allow custom role names. Probably you can use name field for this, but this is a bit unintuitive too

@DarkLight1337 @hmellor

What are your thoughts?

Those are my thoughts for now. @noooop , where do you stand regarding:

1. using transformers' chat_template as the source of truth for the templating?
2. whether truncation limits are a problem, or whether you think we can ignore it? Your experience with LLMs that more frequently rely on chat templates might be useful there.

This is also my concern, which is why I'd like to seek your advice. After all, Sentence Transformers and MTEB are upstream of vLLM, and vLLM only supports a very limited number of CrossEncoder models.

Just to mention

Currently, vLLM does not perform truncation by default, following the OpenAI API behavior for /v1/embeddings.

#24235 (comment)

using transformers' chat_template as the source of truth for the templating?

This makes sense. If the HF Hub repo has an incorrect chat template, you can override in in vLLM via passing --chat-template.

whether truncation limits are a problem, or whether you think we can ignore it? Your experience with LLMs that more frequently rely on chat templates might be useful there.

As @noooop , since we don't allow truncation by default, it should not be a problem.

Ok, I'll modify the PR, so that it uses --chat-template and apply_hf_chat_template from vllm.entrypoints.chat_utils, instead of calling jinja2 directly.

I think that's the right move. I'll also move to chat_template. We should aim for a bit of a format that works conveniently, e.g.

1. Using "query", "document", and perhaps "prompt" for the roles of the messages perhaps? For reference, in Sentence Transformers/MTEB, models are called with nested lists like [["What is the capital of China?", "The capital of China is Beijing."], ...], so I have to know how to convert that to the messages structure.
2. Or assume that message[0]["content"] is the query and message[1]["content"] is the document? This becomes very tricky if you want flexibility in the prompt/instruction as will be possible with Sentence Transformers (model.predict([["What is the capital of China?", "The capital of China is Beijing."], ...], prompt_name="default")). Index 0 for prompts, 1 for query, and 2 for the document is already stronger, but still too arbitrary in my opinion.

Does vLLM support prompts/instructions?

Edit: As mentioned by @Samoed, the above approaches are not very robust to Listwise rerankers which has multiple documents.

• Tom Aarsen

Also, support based on chunk content can be added like

{
  "role": "user",
  "content": [
    {
      "type": "query",
      "text": {  # query/text
        "value": "How does AI work? Explain it in simple terms.",
        "annotations": []
      }
    },
    {
       "type": "document",
       "text": {  # document/text
        "value": "AI works like ...",
      }
  ],
}

But I'm not sure is it possible to handle from jinja and if it work with other libraries

For reference, in Sentence Transformers/MTEB, models are called with nested lists like [["What is the capital of China?", "The capital of China is Beijing."], ...],

By the way, this won't work for ListWise reranking and it's support should be added (created issue in mteb embeddings-benchmark/mteb#3744)

I think we need ask someone from tokenizers/chat template mainaters for better way to handle this

cc @hmellor

Hi @jzakrzew, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

@noooop Sorry, just wanted to clarify one comment, I did not notice you enabled automerge.