Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: nvidia/llama-embed-nemotron-8b
Tasks: AILAStatutes, AfriSentiClassification, AlloProfClusteringS2S.v2, AlloprofReranking, AmazonCounterfactualClassification, ArXivHierarchicalClusteringP2P, ArXivHierarchicalClusteringS2S, ArguAna, ArmenianParaphrasePC, BUCC.v2, BelebeleRetrieval, BibleNLPBitextMining, BigPatentClustering.v2, BiorxivClusteringP2P.v2, BornholmBitextMining, BrazilianToxicTweetsClassification, BulgarianStoreReviewSentimentClassfication, CEDRClassification, CLSClusteringP2P.v2, CSFDSKMovieReviewSentimentClassification, CTKFactsNLI, CataloniaTweetClassification, Core17InstructionRetrieval, CovidRetrieval, CyrillicTurkicLangClassification, CzechProductReviewSentimentClassification, DBpediaClassification, DalajClassification, DiaBlaBitextMining, EstonianValenceClassification, FaroeseSTS, FilipinoShopeeReviewsClassification, FinParaSTS, FinancialPhrasebankClassification, FloresBitextMining, GermanSTSBenchmark, GreekLegalCodeClassification, GujaratiNewsClassification, HALClusteringS2S.v2, HagridRetrieval, IN22GenBitextMining, IndicCrosslingualSTS, IndicGenBenchFloresBitextMining, IndicLangClassification, IndonesianIdClickbaitClassification, IsiZuluNewsClassification, ItaCaseholdClassification, JSICK, KorHateSpeechMLClassification, KorSarcasmClassification, KurdishSentimentClassification, LEMBPasskeyRetrieval, LegalBenchCorporateLobbying, MIRACLRetrievalHardNegatives, MLQARetrieval, MacedonianTweetSentimentClassification, MalteseNewsClassification, MasakhaNEWSClassification, MasakhaNEWSClusteringS2S, MassiveIntentClassification, MedrxivClusteringP2P.v2, MultiEURLEXMultilabelClassification, MultiHateClassification, NTREXBitextMining, NepaliNewsClassification, News21InstructionRetrieval, NollySentiBitextMining, NordicLangClassification, NorwegianCourtsBitextMining, NusaParagraphEmotionClassification, NusaTranslationBitextMining, NusaX-senti, NusaXBitextMining, OdiaNewsClassification, OpusparcusPC, PAC, PawsXPairClassification, PlscClusteringP2P.v2, PoemSentimentClassification, PolEmo2.0-OUT, PpcPC, PunjabiNewsClassification, RTE3, Robust04InstructionRetrieval, RomaniBibleClustering, RuBQReranking, SCIDOCS, SIB200ClusteringS2S, SICK-R, STS12, STS13, STS14, STS15, STS17, STS22.v2, STSB, STSBenchmark, STSES, ScalaClassification, SemRel24STS, SentimentAnalysisHindi, SinhalaNewsClassification, SiswatiNewsClassification, SlovakMovieReviewSentimentClassification, SpartQA, SprintDuplicateQuestions, StackExchangeClustering.v2, StackOverflowQA, StatcanDialogueDatasetRetrieval, SwahiliNewsClassification, SwednClusteringP2P, SwissJudgementClassification, T2Reranking, TERRa, TRECCOVID, Tatoeba, TempReasonL1, ToxicConversationsClassification, TswanaNewsClassification, TweetTopicSingleClassification, TwitterHjerneRetrieval, TwitterURLCorpus, VoyageMMarcoReranking, WebLINXCandidatesReranking, WikiCitiesClustering, WikiClusteringP2P.v2, WikipediaRerankingMultilingual, WikipediaRetrievalMultilingual, WinoGrande, XNLI, indonli

Results for nvidia/llama-embed-nemotron-8b

Model have high performance on these tasks: BrazilianToxicTweetsClassification,DiaBlaBitextMining,FilipinoShopeeReviewsClassification,IndicLangClassification,IndonesianIdClickbaitClassification,KorHateSpeechMLClassification,MIRACLRetrievalHardNegatives,MasakhaNEWSClassification,NollySentiBitextMining,NusaXBitextMining,OpusparcusPC,PAC,PolEmo2.0-OUT,SIB200ClusteringS2S,ScalaClassification,SinhalaNewsClassification,TempReasonL1,WikiClusteringP2P.v2

Tasks with improvements over previous max

1. MIRACLRetrievalHardNegatives +3
2. ScalaClassification +7
3. SIB200ClusteringS2S +3
4. IndonesianIdClickbaitClassification +7

MIRACLRetrievalHardNegatives is in the training set. Scala can be a bit sporadic so wouldn't worry too much about that.

IndonesianIdClickbaitClassification might be a case of a better prompt (the current ones seem quite bad; we should probably update that one). I see that SIB200 also have custom prompt.

@ybabakhin can I ask about the approach that you used for selecting prompts (I see that not all tasks have them).

the current ones seem quite bad; we should probably update that one

We can't update prompts, because results wouldn't be reproducible

@KennethEnevoldsen approach was the following:

• Take prompts from Qwen3-Embedding models: https://github.com/QwenLM/Qwen3-Embedding/blob/main/evaluation/task_prompts.json
  • For example, it already had "IndonesianIdClickbaitClassification": "Given an Indonesian news headlines, classify its into clickbait or non-clickbait",
• Edit a general prompt for Retrieval datasets: from Retrieval the relevant passage for the given query -> Given a question, retrieve passages that answer the question
• Remove datasets which already have own prompt in mteb
• Remove datasets for STS, BitextMining and PairClassification problem types, as they all have general prompts: https://github.com/embeddings-benchmark/mteb/pull/3407/files#diff-cbe979b0366cc5f6a7bbc0f0cf20f494873d162415e68d2b80e0f2bd9a28ce98R437

Thanks for the clarification @ybabakhin. I will merge this in as it doesn't seem like there are any notable changes in the model implementation.

Btw @ybabakhin did you consider running this on RTEB as well? (we can help run the private set if you run the public)

We can't update prompts, because results wouldn't be reproducible

We can introduce a new version of the tasks and rerun existing set of models

@KennethEnevoldsen Thanks for merging!

Btw @ybabakhin did you consider running this on RTEB as well?

Unfortunately, RTEB was released when we already had finalized our data mix. We don't have any Code data in there, while RTEB is pretty code-heavy. So, we're not targeting it with this model.

But we like a Public-Private Leaderboard setup, and we can re-visit RTEB in the future