Skip to content

Number of models should only include "fully evaluated" #4826

Description

@KennethEnevoldsen

Description of the feature

Currently, it seems like there are models evaluated on MMTEB v1, and while that is true, most models are not evaluated on all tasks.

This make it seem like the benchmark is more complete than it is. I would instead display only fully evaluted models

Image

We could add an info box for models in the overview that states "A total of {n} models is evaluated on all tasks. A total of {N} models is evaluated on at least one task within the benchmark."

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestleaderboardissues related to the leaderboard

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions