feat: Add faiss cache backend by Samoed · Pull Request #3402 · embeddings-benchmark/mteb

Samoed · 2025-10-17T12:28:47Z

Added FAISS to cache backend. Now CacheEmbeddingsWrapper can support multiple backends that can be passed directly.

from mteb.models.cache_wrappers.cache_backends.faiss_cache import FaissCache

model = mteb.get_model(...)
cachedmodel = CachedEmbeddingWrapper(model, cache_dir, cache_backend=FaissCache)

Probably something similar can be done for Retrieval tasks

# Conflicts: # mteb/models/cache_wrappers/cache_wrapper.py

KennethEnevoldsen

great changes - only a few minor things

KennethEnevoldsen · 2025-10-17T13:34:11Z

+logger = logging.getLogger(__name__)
+
+
+class VectorCacheMap:


Should we rename this to NumpyCache?

KennethEnevoldsen · 2025-10-17T13:34:50Z

+
+
+@runtime_checkable
+class CacheBackendProtocol(Protocol):


Loving it - this makes it very easy to change + test

+1, this is very nice!

KennethEnevoldsen · 2025-10-17T13:38:11Z

-                self.cache_dict[task_name] = _VectorCacheMap(
-                    self.cache_path / task_name
-                )
+                self.cache_dict[task_name] = self.backend(self.cache_path / task_name)


So, a whole new cache PR task - I am fine with this, but it prevents re-use across

Yes, agree. Can remove it

does the check include prompt (if not we probably have to handle that)

Yes, maybe this is the reason behind different indexes for different tasks. I think the simplest solution would be to keep cache per task to ensure that prompts will apply correctly, because they can change in model.encode

yeah propably the best choice

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

orionw

Looks great! Excited to have this in 🙌

orionw · 2025-10-17T16:55:31Z

 xet = ["huggingface_hub>=0.32.0"]
 youtu = ["tencentcloud-sdk-python-common>=3.0.1454", "tencentcloud-sdk-python-lkeap>=3.0.1451"]
+faiss-cpu = ["faiss-cpu>=1.12.0"]
+faiss-gpu = ["faiss-gpu>=1.12.0"]


Did you test with cpu and gpu? It's been a minute since I used them, but IIRC the gpu one has some quirks.

I faiss-gpu is deprecated on pypi, but exists only on conda. I'll leave only cpu version then

orionw · 2025-10-17T16:55:45Z

+
+
+@runtime_checkable
+class CacheBackendProtocol(Protocol):


+1, this is very nice!

Samoed · 2025-10-19T09:49:31Z

@KennethEnevoldsen Is it good to merge?

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

# Conflicts: # pyproject.toml

Samoed added 7 commits October 17, 2025 13:14

move cache wrapper to different folder

66817b7

upd docstring

7052819

add missing init

9b1cffc

use model's similarity functions

80c8470

add faiss cache backend

4f43541

Merge branch 'v2.0.0' into move_cache_wrapper

dca7718

# Conflicts: # mteb/models/cache_wrappers/cache_wrapper.py

add faiss cache backend

28fb362

Samoed requested review from KennethEnevoldsen and orionw October 17, 2025 12:28

KennethEnevoldsen reviewed Oct 17, 2025

View reviewed changes

Comment thread mteb/models/cache_wrappers/cache_wrapper.py Outdated

Samoed and others added 3 commits October 17, 2025 17:00

Update mteb/models/cache_wrappers/cache_wrapper.py

981d7de

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

update cache index

5105641

add faiss to tests

4946b55

orionw approved these changes Oct 17, 2025

View reviewed changes

Samoed added 2 commits October 17, 2025 20:32

remove faiss gpu

2e60fee

upd docs

219fd2e

KennethEnevoldsen approved these changes Oct 19, 2025

View reviewed changes

Comment thread docs/advanced_usage/cache_embeddings.md Outdated

Samoed and others added 3 commits October 19, 2025 23:31

Update docs/advanced_usage/cache_embeddings.md

06000cb

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

Merge branch 'v2.0.0' into move_cache_wrapper

7a86af3

# Conflicts: # pyproject.toml

update init

76c1701

Samoed enabled auto-merge (squash) October 19, 2025 20:55

Samoed merged commit aa88b98 into v2.0.0 Oct 19, 2025
11 checks passed

Samoed deleted the move_cache_wrapper branch October 19, 2025 21:11

		logger = logging.getLogger(__name__)


		class VectorCacheMap:



		@runtime_checkable
		class CacheBackendProtocol(Protocol):



		@runtime_checkable
		class CacheBackendProtocol(Protocol):

Uh oh!

Conversation

Samoed commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KennethEnevoldsen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

orionw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Samoed Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Samoed commented Oct 19, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Samoed commented Oct 17, 2025 •

edited

Loading

Samoed Oct 17, 2025 •

edited

Loading