When pointing embeddings-benchmark/mteb#2035 to this branch, it seems like MIEB results cannot be displayed due to "Number of parameters".

@gowitheflow-1998 @KennethEnevoldsen here's a screenshot of the LB, hacked to point to this branch. eng and lite versions were able to render as well. Cache needed to be wiped.

there's a few task where the main metric was wrong when we implemented them and isn't matching with the paper. Let me double-check all tasks and get back. Might be a good idea to replace the scores in main metric with the actual main metrics before we merge I think

Also seems like the performance v. model size plot need some model references. You can add these in:

mteb.leaderboard.figures.models_to_annotate which is currently:

models_to_annotate = [
    "all-MiniLM-L6-v2",
    "GritLM-7B",
    "LaBSE",
    "multilingual-e5-large-instruct",
]

Also seems like the performance v. model size plot need some model references. You can add these in:

mteb.leaderboard.figures.models_to_annotate which is currently:

models_to_annotate = [
    "all-MiniLM-L6-v2",
    "GritLM-7B",
    "LaBSE",
    "multilingual-e5-large-instruct",
]

What does "some model reference" mean? How do we select the models for this list?

Figured it out 👍

[update] Added a few models that ranked first from a few task types:

• "EVA02-CLIP-bigE-14-plus"
• "voyage-multimodal-3"
• "e5-v"
• "VLM2Vec-Full"

The performance per task type plot isn't showing though 🤔 says it only contains one task type when there are 8.

hmm not sure why this is happening - @x-tabdeveloping do you have an idea?

I'll have a look at it tomorrow

@isaac-chung My guess would be it's cause of mteb.leaderboard.figures.task_types:

task_types = [
    "BitextMining",
    "Classification",
    "MultilabelClassification",
    "Clustering",
    "PairClassification",
    "Reranking",
    "Retrieval",
    "STS",
    "Summarization",
    # "InstructionRetrieval",
    # Not displayed, because the scores are negative,
    # doesn't work well with the radar chart.
    "Speed",
]

The reason I made this list was because instruction retrieval shows scores in the negatives and that doesn't really work with the radar chart.
We could either extend this list or just make a list with the exceptions and infer the list of task types from somewhere else.

@isaac-chung My guess would be it's cause of mteb.leaderboard.figures.task_types:

task_types = [
    "BitextMining",
    "Classification",
    "MultilabelClassification",
    "Clustering",
    "PairClassification",
    "Reranking",
    "Retrieval",
    "STS",
    "Summarization",
    # "InstructionRetrieval",
    # Not displayed, because the scores are negative,
    # doesn't work well with the radar chart.
    "Speed",
]

The reason I made this list was because instruction retrieval shows scores in the negatives and that doesn't really work with the radar chart. We could either extend this list or just make a list with the exceptions and infer the list of task types from somewhere else.

That's it. Thanks! It's working now.

have fixed main metric issue by overwriting main scores with actual main metric scores; deleted previous incomplete Jina runs with a old version that only has a few task results.

overwritten scores include:

task_metric_mapping = {"BLINKIT2IRetrieval.json":"cv_recall_at_1",
 "BLINKIT2TRetrieval.json":"cv_recall_at_1",
 "ImageCoDeT2IRetrieval.json":"cv_recall_at_3",
 "ROxfordEasyI2IMultiChoice.json":"map_at_5",
 "ROxfordMediumI2IMultiChoice.json":"map_at_5",
 "ROxfordHardI2IMultiChoice.json":"map_at_5",
 "RParisEasyI2IMultiChoice.json":"map_at_5",
 "RParisMediumI2IMultiChoice.json":"map_at_5",
 "RParisHardI2IMultiChoice.json":"map_at_5", 
 "TinyImageNetClustering.json":"nmi",
 "CIFAR10Clustering.json":"nmi",
 "CIFAR100Clustering.json":"nmi",
 "ImageNet10Clustering.json":"nmi",
 "ImageNetDog15Clustering.json":"nmi",
}

@gowitheflow-1998 good stuff! Are we ready to merge?

@gowitheflow-1998 good stuff! Are we ready to merge?

yeah, merged! adding @Muennighoff as co-author for running most of the results here!

Does this have everything from https://github.com/embeddings-benchmark/tmp i.e. we can safely delete that repo?

Does this have everything from https://github.com/embeddings-benchmark/tmp i.e. we can safely delete that repo?

yeah! all results are here