Hi 👋 I'd like to add a few more tasks that we've prepared for Spanish (and multilingual) embedding evaluation:

Reranking:

• MIRACLReranking (https://huggingface.co/datasets/miracl/miracl): This is originally retrieval dataset, here formulated as a multilingual reranking task. It contains a corpus of passages from Wikipedia articles and human-annotated relevance judgements for some of them. We only took passages where relevance judgements are available since those are suitable for an MTEB re-ranking task.

Retrieval:

• MIRACLRetrieval (https://huggingface.co/datasets/miracl/miracl): Same as above, but as a multilingual retrieval task.
• XMarket (https://xmrec.github.io/): An e-commerce category-to-product retrieval dataset in multiple languages.
• SpanishPassageRetrieval(S2S and S2P) (https://mklab.iti.gr/results/spanish-passage-retrieval-dataset/): A test collection for passage retrieval from health-related Web resources in Spanish.
• MintakaRetrieval (https://github.com/amazon-science/mintaka): A multilingual end-to-end question answering dataset.
• XPQARetrieval (https://arxiv.org/abs/2305.09249): xPQA is a large-scale annotated cross-lingual Product QA dataset.

Clustering:

• SpanishNewsClusteringP2P (https://www.kaggle.com/datasets/kevinmorgado/spanish-news-classification): Clustering of Spanish news articles, 7 topics in total.
• FloresClusteringS2S (https://huggingface.co/datasets/facebook/flores): Clustering of Spanish sentences from various web articles, 32 topics in total.

STS:

• STSES (https://huggingface.co/datasets/PlanTL-GOB-ES/sts-es): STS task prepared from Spanish test sets from SemEval-2014 (Agirre et al., 2014) and SemEval-2015 (Agirre et al., 2015).