Skip to content

fix filter by model size#4794

Merged
Samoed merged 1 commit into
mainfrom
fix_filter
Jun 10, 2026
Merged

fix filter by model size#4794
Samoed merged 1 commit into
mainfrom
fix_filter

Conversation

@Samoed

@Samoed Samoed commented Jun 10, 2026

Copy link
Copy Markdown
Member

Close #4793

mteb/leaderboard/app.py:594-596 initialized default_languages from the polars language column (full codes like "eng-Latn"), while every consumer (update_scores_on_lang_change, update_tables, and _cache_on_benchmark_select) treats lang_select values as 3-letter codes
("eng").

Consequences in update_tables:

  1. set(languages).issuperset(_benchmark_full_languages(...)) always returned False (full vs 3-letter), so the "skip when all selected" optimization never kicked in.
  2. The mask pl.element().str.split("-").list.first().is_in(lang_set) split each row's "eng-Latn" to "eng" and looked it up against the full-code lang_set — matching zero rows.

Initial render dodged it (no update_tables); any filter click triggered the broken language filter → empty table.

@Samoed Samoed added the leaderboard issues related to the leaderboard label Jun 10, 2026
@Samoed Samoed enabled auto-merge (squash) June 10, 2026 16:49
@Samoed Samoed merged commit dc31aec into main Jun 10, 2026
12 checks passed
@Samoed Samoed deleted the fix_filter branch June 10, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

leaderboard issues related to the leaderboard

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Specifying any filter under Advanced Model Filters completely breaks leaderboard

1 participant