Skip to content

[fix] Collapse single-key multimodal dicts to bare modality#3779

Merged
tomaarsen merged 1 commit into
huggingface:mainfrom
tomaarsen:worktree-mteb-bge-vl-issue
May 20, 2026
Merged

[fix] Collapse single-key multimodal dicts to bare modality#3779
tomaarsen merged 1 commit into
huggingface:mainfrom
tomaarsen:worktree-mteb-bge-vl-issue

Conversation

@tomaarsen

@tomaarsen tomaarsen commented May 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Hello!

Pull Request overview

  • Treat single-key multimodal dicts (e.g. {"image": pil}) as equivalent to bare modality inputs

Details

MTEB passes image inputs as {"image": pil} dicts when running vision-retrieval benchmarks (see embeddings-benchmark/mteb#4310 (comment)). These were previously classified as the tuple modality ("image",) and routed through the multimodal path, which fails the supports() check for models that only declare "image" support (e.g. BGE-VL).

With this PR, infer_modality collapses a single-key multimodal dict to the bare modality string. {"image": pil} now returns "image" instead of ("image",). parse_inputs and pair_to_messages correspondingly unwrap the dict so downstream sees the raw value, matching what they would have seen for a bare PIL input.

The existing {"audio": ...} / {"video": ...} wrapper tests are updated to expect the bare modality (they were already going through the same unwrap path), and new tests cover parse_inputs and pair_to_messages for the image/text cases that the previous tuple-based handling silently masked.

Notably, this fixes:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/BGE-VL-base', trust_remote_code=True)
embedding = model.encode({"image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ettin-reranker/mteb_ndcg10_all-MiniLM-L6-v2.png"})
print(embedding)
Traceback (most recent call last):
  File "c:\code\sentence-transformers\demo_bge_vl.py", line 5, in <module>
    embedding = model.encode({"image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ettin-reranker/mteb_ndcg10_all-MiniLM-L6-v2.png"})
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\torch\utils\_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\util\decorators.py", line 41, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\sentence_transformer\model.py", line 650, in encode
    features = self.preprocess(inputs_batch, prompt=prompt, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\base\model.py", line 542, in preprocess
    raise ValueError(message)
ValueError: Modality 'image' is not supported by this SentenceTransformer model. Supported modalities: text, image, image+text
This model supports image individually, but not in the same input. Please process each modality separately.
  • Tom Aarsen

@tomaarsen tomaarsen requested a review from Copilot May 19, 2026 16:24
@tomaarsen tomaarsen mentioned this pull request May 19, 2026
Merged
6 tasks
Copilot AI reviewed May 19, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes single-key multimodal dicts like {"image": pil} behave equivalently to bare modality inputs (e.g., a raw PIL image). Previously, such dicts were classified as the tuple modality ("image",) and routed through the multimodal path, breaking supports() checks for models that only declare single-modality support (e.g., BGE-VL via MTEB).

Changes:

  • infer_modality now collapses 1-key multimodal dicts to the bare modality string, and parse_inputs / _to_content unwrap the dict so downstream sees the raw value.
  • Introduced MULTIMODAL_DICT_KEYS constant in modality_types.py to centralize valid modality keys.
  • Updated existing audio/video wrapper tests to expect the bare modality, and added new tests for image/text single-key dicts in infer_modality, parse_inputs, and pair_to_messages.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
sentence_transformers/base/modality_types.py Adds the MULTIMODAL_DICT_KEYS frozenset constant.
sentence_transformers/base/modality.py Collapses 1-key dicts in infer_modality; unwraps the dict in parse_inputs and _to_content.
tests/base/test_modality.py Updates existing wrapper-modality assertions and adds new tests for image/text single-key handling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tomaarsen tomaarsen merged commit 5ce97d6 into huggingface:main May 20, 2026
22 checks passed
tomaarsen added a commit that referenced this pull request May 20, 2026
@tomaarsen
[fix] Collapse single-key multimodal dicts to bare modality (#3779)
610a7c5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tomaarsen