feat: Add ColBERT late interaction model support by ieBoytsov · Pull Request #33686 · vllm-project/vllm

ieBoytsov · 2026-02-03T11:01:48Z

Purpose

Add support for ColBERT late interaction models in vLLM. ColBERT is a retrieval model that uses per-token embeddings with MaxSim scoring, offering a balance between the accuracy of cross-encoders and the efficiency of bi-encoders.

Resolves #13827

Test Plan

Offline tests (embedding correctness, MaxSim scoring, HF comparison)

pytest tests/models/language/pooling/test_colbert.py -v

Online tests (rerank, score, pooling endpoints)

pytest tests/entrypoints/pooling/score/test_online_colbert.py -v

Test Result

All tests passed. Tests include:

Comparison with embeddings from original HuggingFace model
Relevance ordering validation
MaxSim scoring correctness (1:1, 1:N, N:N configurations)
Various other cases for both offline and online scoring

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

github-actions · 2026-02-03T11:01:58Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

gemini-code-assist

Code Review

This pull request introduces support for ColBERT late interaction models, a valuable addition to vLLM's capabilities. The implementation is well-structured, spanning model configuration, entrypoints for both online and offline inference, and the model definition itself. The inclusion of comprehensive online and offline tests is commendable, ensuring the correctness and robustness of the new feature. My primary feedback concerns code duplication in the MaxSim scoring logic, which appears in both llm.py and serving.py. Consolidating this into a shared utility function would enhance the code's maintainability. Overall, this is a strong and well-executed contribution.

vllm/entrypoints/llm.py

vllm/entrypoints/pooling/score/serving.py

mergify · 2026-02-03T11:06:10Z

Hi @ieBoytsov, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

tugot17 · 2026-02-03T11:09:53Z

Awesome work Ilya 🤗🤗

mergify · 2026-02-03T12:49:48Z

Hi @ieBoytsov, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2026-02-03T13:01:46Z

Hi @ieBoytsov, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

noooop · 2026-02-03T13:10:03Z

Awesome work!

Please add an example to this awesome feature to show users how to use it.
https://github.com/vllm-project/vllm/tree/main/examples/pooling/score
Please add some documentation to introduce this awesome feature.
https://github.com/vllm-project/vllm/blob/main/docs/models/pooling_models.md

vllm/model_executor/models/registry.py

mergify · 2026-02-03T15:15:16Z

Documentation preview: https://vllm--33686.org.readthedocs.build/en/33686/

vllm/entrypoints/pooling/score/serving.py

ieBoytsov · 2026-02-03T15:18:35Z

Hi @noooop thanks for your comments, made a separate commit with docs, examples and minor fixes!

mergify · 2026-02-04T13:31:34Z

Hi @ieBoytsov, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

vllm/entrypoints/pooling/score/utils.py

Signed-off-by: Ilya Boytsov <ilyaboytsov1805@gmail.com>

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Ilya Boytsov <boytsovpanamera@mail.ru>

noooop · 2026-02-04T14:32:37Z

waiting #33794 for fix main fail.

ieBoytsov · 2026-02-04T17:12:31Z

I see issues in the CI. Working on it :)

Signed-off-by: Ilya Boytsov <ilyaboytsov1805@gmail.com>

ieBoytsov · 2026-02-04T17:25:49Z

tests/models/registry.py

-    "ColBERTModel": _HfExamplesInfo(
-        "answerdotai/answerai-colbert-small-v1",
-    ),
+    "HF_ColBERT": _HfExamplesInfo("answerdotai/answerai-colbert-small-v1"),


I left only HF_ColBERT naming in the registry because it is from the original model https://huggingface.co/colbert-ir/colbertv2.0 and is also used in another model that we use in tests https://huggingface.co/answerdotai/answerai-colbert-small-v1

ieBoytsov · 2026-02-04T21:13:55Z

Hi @noooop I’ve pushed a small fix, could you please re-enable auto-merge or merge when you have a moment? Thanks in advance :)

Signed-off-by: Ilya Boytsov <ilyaboytsov1805@gmail.com> Signed-off-by: Ilya Boytsov <boytsovpanamera@mail.ru> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: felix01.yu <felix01.yu@vipshop.com>

Signed-off-by: Ilya Boytsov <ilyaboytsov1805@gmail.com> Signed-off-by: Ilya Boytsov <boytsovpanamera@mail.ru> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>

ieBoytsov requested review from ProExpertProg, WoosukKwon, aarnphm, chaunceyjiang, hmellor, houseroad, mgoin, noooop, robertgshaw2-redhat, tlrmchlsmth, yewentao256 and youkaichao as code owners February 3, 2026 11:01

mergify bot added frontend new-model Requests to new models labels Feb 3, 2026

gemini-code-assist bot reviewed Feb 3, 2026

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/pooling/score/serving.py Outdated Show resolved Hide resolved

ieBoytsov force-pushed the add-colbert-support branch from e4db81a to 8cc0a36 Compare February 3, 2026 12:45

ieBoytsov force-pushed the add-colbert-support branch from 8cc0a36 to c6c49ba Compare February 3, 2026 12:57

noooop reviewed Feb 3, 2026

View reviewed changes

vllm/model_executor/models/registry.py Outdated Show resolved Hide resolved

ieBoytsov requested review from DarkLight1337 and ywang96 as code owners February 3, 2026 15:14

mergify bot added the documentation Improvements or additions to documentation label Feb 3, 2026

ieBoytsov commented Feb 3, 2026

View reviewed changes

vllm/entrypoints/pooling/score/serving.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Feb 4, 2026

View reviewed changes

vllm/entrypoints/pooling/score/utils.py Outdated Show resolved Hide resolved

Infer ColBERT dimension from weights automatically

9b1743a

Signed-off-by: Ilya Boytsov <ilyaboytsov1805@gmail.com>

ieBoytsov force-pushed the add-colbert-support branch from 96587fd to 9b1743a Compare February 4, 2026 13:57

Update vllm/entrypoints/pooling/score/utils.py

0b92f20

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Ilya Boytsov <boytsovpanamera@mail.ru>

noooop added ready ONLY add when PR is ready to merge/full CI is needed and removed ready ONLY add when PR is ready to merge/full CI is needed labels Feb 4, 2026

Merge branch 'main' into add-colbert-support

7a3af87

noooop enabled auto-merge (squash) February 4, 2026 15:56

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 4, 2026

noooop approved these changes Feb 4, 2026

View reviewed changes

fix model registry

570189e

Signed-off-by: Ilya Boytsov <ilyaboytsov1805@gmail.com>

auto-merge was automatically disabled February 4, 2026 17:23
Head branch was pushed to by a user without write access

Merge branch 'main' into add-colbert-support

f040995

ieBoytsov commented Feb 4, 2026

View reviewed changes

Merge branch 'main' into add-colbert-support

ecd13fa

noooop merged commit 439afa4 into vllm-project:main Feb 5, 2026
52 checks passed

ieBoytsov mentioned this pull request Feb 9, 2026

Extend ColBERT support to non-standard BERT backbones #34170

Merged

ieBoytsov mentioned this pull request Mar 19, 2026

[Model] Add LFM2-ColBERT-350M support #37528

Merged

5 tasks

Uh oh!

Conversation

ieBoytsov commented Feb 3, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

github-actions bot commented Feb 3, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Feb 3, 2026

Uh oh!

tugot17 commented Feb 3, 2026

Uh oh!

mergify bot commented Feb 3, 2026

Uh oh!

mergify bot commented Feb 3, 2026

Uh oh!

noooop commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Feb 3, 2026

Uh oh!

Uh oh!

ieBoytsov commented Feb 3, 2026

Uh oh!

mergify bot commented Feb 4, 2026

Uh oh!

Uh oh!

noooop commented Feb 4, 2026

Uh oh!

ieBoytsov commented Feb 4, 2026

Uh oh!

ieBoytsov Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

ieBoytsov commented Feb 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ieBoytsov commented Feb 3, 2026 •

edited by github-actions bot

Loading

noooop commented Feb 3, 2026 •

edited

Loading