Skip to content

model: add Verm1ion/ColTurk-VDR-Qwen3VL-4B-v1.0#4796

Merged
KennethEnevoldsen merged 2 commits into
embeddings-benchmark:mainfrom
Verm1lion:Verm1lion-patch-1
Jun 11, 2026
Merged

model: add Verm1ion/ColTurk-VDR-Qwen3VL-4B-v1.0#4796
KennethEnevoldsen merged 2 commits into
embeddings-benchmark:mainfrom
Verm1lion:Verm1lion-patch-1

Conversation

@Verm1lion

@Verm1lion Verm1lion commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Adds ColTurk-VDR-Qwen3VL-4B-v1.0 — a ColBERT-style late-interaction visual document retriever built on Qwen/Qwen3-VL-4B-Instruct (LoRA r=32, published as merged full weights; colpali-engine ColQwen3 architecture, Apache-2.0, open weights, public training code and data).

Since the model is a colpali-engine-native checkpoint (not a trust_remote_code repo), this PR also adds a small ColQwen3EngineWrapper that mirrors the existing ColQwen2_5Wrapper pattern (delegates to colpali_engine.models.ColQwen3 / ColQwen3Processor via ColPaliEngineWrapper).

ViDoRe V3 results (8 public retrieval tasks, full corpus, all queries, MaxSim) — mean NDCG@10 = 0.5584:

Task NDCG@10
Vidore3ComputerScienceRetrieval 0.7306
Vidore3EnergyRetrieval 0.6238
Vidore3PharmaceuticalsRetrieval 0.6156
Vidore3FinanceEnRetrieval 0.5851
Vidore3HrRetrieval 0.5463
Vidore3IndustrialRetrieval 0.4624
Vidore3PhysicsRetrieval 0.4564
Vidore3FinanceFrRetrieval 0.4467

Eval code & raw result JSONs (with seeded bootstrap 95% CIs): https://github.com/Verm1lion/ColTurk-VDR — results PR: embeddings-benchmark/results#565 · private-split request: #4797.

  • I have filled out the ModelMeta object to the extent possible
  • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision) and
    • mteb.get_model_meta(model_name, revision)
  • I have tested the implementation works on a representative set of tasks.
  • The model is public, i.e., is available either as an API or the weights are publicly available to download
  • I reproduced results from the original paper (if applicable) on at least one benchmark, and I am including the results in the PR description.

Notes for reviewers:

  • The wrapper's load path (ColQwen3.from_pretrained + ColQwen3Processor.from_pretrained on the published repo) was smoke-verified end-to-end on the exact revision pinned in the ModelMeta.
  • The results above were produced with the published self-contained harness (identical loading + MaxSim over the full corpora, all queries). Happy to re-run anything via mteb.evaluate on request.
  • Training data (the ColPali train set) vs. ViDoRe V3 contamination was checked empirically (perceptual-hash scan over train images x V3 corpora: 0 exact duplicates; report in the linked repo).

release_date="2026-06-11",
modalities=["image", "text"],
n_parameters=4_505_515_136,
n_embedding_parameters=None,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
n_embedding_parameters=None,
n_embedding_parameters=388_956_160,

Comment thread mteb/models/model_implementations/colqwen_models.py
@Verm1lion

Copy link
Copy Markdown
Contributor Author

Applied both suggestions, thanks for the quick review!

@KennethEnevoldsen KennethEnevoldsen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any outstanding issues

@KennethEnevoldsen KennethEnevoldsen enabled auto-merge (squash) June 11, 2026 18:50
@KennethEnevoldsen KennethEnevoldsen merged commit 7ea3685 into embeddings-benchmark:main Jun 11, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants