Yeah ! 🚀 I made the change, do you need the json results ?

Yes that would be great!

Hey @Muennighoff, do you need the results to be merged in this repository https://huggingface.co/datasets/mteb/results, or can we juste give you the json files we have ?

Yeah feel free to directly open a PR there e.g. like this one: #218 (comment)

You just need to create a folder for each model and place the files in there; You can clone the repo with git clone ... ; instructions are also here https://huggingface.co/datasets/mteb/results/discussions?new_pr=true

I think we are good with the changes, and the results are there : https://huggingface.co/datasets/mteb/results/discussions/28

Yeah ! 🚀 I made the change, do you need the json results ?

Yes that would be great!

Hey @Muennighoff, do you need the results to be merged in this repository https://huggingface.co/datasets/mteb/results, or can we juste give you the json files we have ?

Yeah feel free to directly open a PR there e.g. like this one: #218 (comment)
You just need to create a folder for each model and place the files in there; You can clone the repo with git clone ... ; instructions are also here https://huggingface.co/datasets/mteb/results/discussions?new_pr=true

I think we are good with the changes, and the results are there : https://huggingface.co/datasets/mteb/results/discussions/28

Amazing! I think this one would still be nice #218 (comment) if okay with you but other than that we can merge both!

Amazing! I think this one would still be nice #218 (comment) if okay with you but other than that we can merge both!

@Muennighoff we made the changes for #218 (comment), can you please have a look at AbsTaskClustering, MasakhaNEWSClusteringP2P and MasakhaNEWSClusteringS2S files ?

Yes let's merge ! 💯

Still need to update MasakhaNEWS files in https://huggingface.co/datasets/mteb/results/discussions/28 I think & then I will add a French leaderboard tab if you want! 🙌 Should I just link to your GitHub accounts / this PR for the Credits of the new tabs or are you planning to write a paper on it?

I think the leaderboard(s) would consist of the following:

TASK_LIST_CLASSIFICATION_FR = [
    "AmazonReviewsClassification (fr)",
    "MasakhaNEWSClassification (fra)",
    "MassiveIntentClassification (fr)",
    "MassiveScenarioClassification (fr)",
]

TASK_LIST_CLUSTERING_FR = [
    "AlloProfClusteringP2P",
    "AlloProfClusteringS2S",
    "HALClusteringS2S",
    "MLSUMClusteringP2P",
    "MLSUMClusteringS2S",
    "MasakhaNEWSClusteringP2P (fra)",
    "MasakhaNEWSClusteringS2S (fra)",
]

TASK_LIST_PAIR_CLASSIFICATION_FR = [
    "OpusparcusPC (fr)",
    "PawsX (fr)",
]

TASK_LIST_RERANKING_FR = [
    "AlloprofReranking",
    "SyntecReranking",
]
TASK_LIST_RETRIEVAL_FR = [
    "AlloprofRetrieval",
    "BSARDRetrieval",
    "MintakaRetrieval",
    "MultiLongDocRetrieval",
    "SyntecRetrieval",
    "XPQARetrieval",
]
TASK_LIST_STS_FR = [
    "STS22 (fr)",
    "STSBenchmarkMultilingualSTS (fr)",
    "SICKFr",
]
TASK_LIST_SUMMARIZATION_FR = ["SummEvalFr"]

It's a total of 25 so also enough for an overall leaderboard tab imo.
Let me know if I am missing anything!

Also I added a few other French tasks like xPQA etc. not from this PR - would be great if you could eval on them too 🙌

We will re-run the evaluations on MasakhaNEWS, we will also run evals on the datasets you added. The PR will be updated soon.

For the Credits, we have a paper that is under review for the moment. You can still add our GitHub accounts maybe ?

Concerning the list of tasks for the leaderboard, I think you can add these two tasks in TASK_LIST_CLASSIFICATION: "MTOPDomainClassification", "MTOPIntentClassification".

We will re-run the evaluations on MasakhaNEWS, we will also run evals on the datasets you added. The PR will be updated soon.

Amazing!

For the Credits, we have a paper that is under review for the moment. You can still add our GitHub accounts maybe ?

Cool! Happy to help if I can be useful! I will put your GitHubs in the meantime 👍

Concerning the list of tasks for the leaderboard,

Good catch - 27 datasets then!

@Muennighoff do you need models charcteristics like their size, embedding dim and max token length for the leaderboard ? We have already gathered them in these two files (CSV or JSON, you choose what suits you most):
JSON: https://github.com/Lyon-NLP/mtebscripts/blob/main/script_mteb_french/results_analysis/model_specs.json
CSV: https://github.com/Lyon-NLP/mtebscripts/blob/main/script_mteb_french/results_analysis/models_characteristics.csv

Do you also want to run the French split of MultiLongDocRetrieval and include it? I saw it wasn't included in the result files

I have preliminary added the French leaderboard: https://huggingface.co/spaces/mteb/leaderboard 🇫🇷🚀
Let me know if we should change anything / feel free to open a PR! Also would be cool to still include MultiLongDocRetrieval I think :)

Edit: A few models still missing, updating....

I have preliminary added the French leaderboard: https://huggingface.co/spaces/mteb/leaderboard 🇫🇷🚀 Let me know if we should change anything / feel free to open a PR! Also would be cool to still include MultiLongDocRetrieval I think :)

That's so great, thanks ! 🚀 Just one small thing, could you please add @schmarion in the credits section ?

We'll check MultiLongDocRetrieval and will open a PR with the results soon.

Edit: yes some models and a dataset (SummEvalFr) are missing

Thanks a lot ! So good to finally see the results there 🤩 . Could you please rebind my name in the credits. I happen to have changed my account name recently and the link is broken... @MathieuCiancone

Hi guys! I just want to congratulate everybody who has worked on this! You made my day!

I have preliminary added the French leaderboard: https://huggingface.co/spaces/mteb/leaderboard 🇫🇷🚀 Let me know if we should change anything / feel free to open a PR! Also would be cool to still include MultiLongDocRetrieval I think :)

Edit: A few models still missing, updating....

I looked into the dataset and it is actually quite huge (not even talking about the English subset 😅). Even for text embedding API (which tend to be cheap), embedding the whole corpus would lead to significant API costs (including openai 3 embedding model, mistral, voyage and cohere). I think we will save this for smaller datasets, as a greater variety of datasets is more insightful that bigger datasets... 🤔

I looked into the dataset and it is actually quite huge (not even talking about the English subset 😅). Even for text embedding API (which tend to be cheap), embedding the whole corpus would lead to significant API costs (including openai 3 embedding model, mistral, voyage and cohere). I think we will save this for smaller datasets, as a greater variety of datasets is more insightful that bigger datasets... 🤔

So for French, it's 20 queries & 10K docs w/ 9.6K avg characters per doc. I think it's comparable to TREC-COVID which is 171K docs but only ~900 avg characters per doc. TREC-COVID is one of the cheapest English Retrieval datasets. Also for many models that cannot handle the long sequence length of 9.6K characters it'll be cheaper. But ofc up to you, no worries if you don't want to include it 👍