Skip to content

cli.md

Latest commit

 

History

History
93 lines (63 loc) · 3.01 KB

cli.md

File metadata and controls

93 lines (63 loc) · 3.01 KB
title Command Line
icon lucide/terminal

Command Line

This described the is the command line interface for mteb.

mteb is a toolkit for evaluating the quality of embedding models on various benchmarks. It supports the following commands:

In the following we outline some sample use cases, but if you want to learn more about the arguments for each command you can run:

mteb {command} --help

Running Models on Tasks

To run a model on a set of tasks, use the mteb run command. For example:

mteb run -m sentence-transformers/average_word_embeddings_komninos \
         -t Banking77Classification EmotionClassification \
         --output-folder mteb_output

This will create a folder mteb_output/{model_name}/{model_revision} containing the results of the model on the specified tasks supplied as a json file; {task_name}.json.

Listing Available Tasks

To list the available tasks within MTEB, use the mteb available-tasks command. For example:

mteb available-tasks # list _all_ available tasks

You can also use the multiple arguments for filtering:

mteb available-tasks --task-types Retrieval --languages eng # list all English (eng) retrieval tasks

Listing Available Benchmarks

To list the available benchmarks within MTEB:

mteb available-benchmarks # list all available benchmarks

Creating Model Metadata

Once a model is run you can create the metadata for a model card from a folder of results, use the mteb create-meta command. For example:

mteb create-meta --results-folder mteb_output/sentence-transformers__average_word_embeddings_komninos/{revision} \
                 --output-path model_card.md

This will create a model card at model_card.md containing the metadata for the model on MTEB within the YAML frontmatter. This will make the model discoverable on the MTEB leaderboard.

Running the Leaderboard

To run the MTEB leaderboard locally, use the mteb leaderboard command. For example:

mteb leaderboard

You can specify a custom cache path and other options:

mteb leaderboard --cache-path results --port 8080 --share

Available options:

  • --cache-path PATH: Custom path for model results cache
  • --host HOST: Host to run the server on (default: 0.0.0.0)
  • --port PORT: Port to run the server on (default: 7860)
  • --share: Create a public URL for the leaderboard
  • --rebuild: Force rebuild from full results repository, bypassing cached JSON

For more details on running the leaderboard, see the leaderboard documentation.