Skip to content

[v2] reupload reranking datasets in old format#2097

Merged
Samoed merged 13 commits into
v2.0.0from
reupload_retrieval_datasets
Feb 24, 2025
Merged

[v2] reupload reranking datasets in old format#2097
Samoed merged 13 commits into
v2.0.0from
reupload_retrieval_datasets

Conversation

@Samoed

@Samoed Samoed commented Feb 18, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

Closes #1878

  • "MindSmallReranking",
  • "SciDocsRR",
  • "StackOverflowDupQuestions",
  • "WebLINXCandidatesReranking",
  • "AlloprofReranking",
  • "SyntecReranking",
  • "VoyageMMarcoReranking",
  • "ESCIReranking",
  • "MIRACLReranking",
  • "WikipediaRerankingMultilingual",
  • "RuBQReranking",
  • "T2Reranking",
  • "MMarcoReranking",
  • "CMedQAv1-reranking",
  • "CMedQAv2-reranking",
  • "NamaaMrTydiReranking",

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

@orionw Can you review, please?

@Samoed Samoed added the v2 label Feb 18, 2025
@Samoed Samoed marked this pull request as ready for review February 20, 2025 15:22
@Samoed Samoed requested a review from orionw February 20, 2025 15:22
@Samoed Samoed mentioned this pull request Feb 20, 2025
Merged
4 tasks
@Samoed Samoed mentioned this pull request Feb 21, 2025
Merged
4 tasks
orionw
orionw approved these changes Feb 21, 2025
View reviewed changes

@orionw orionw left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great @Samoed, thanks for finalizing this.

My only question is does this affect the listing of tasks? e.g. what distinguishes reranking vs retrieval now, when you load with the mteb.load_tasks for reranking only? I would assume we need some mapping now to maintain that they are reranking tasks.

EDIT: perhaps that's in the task meta? If so then I have no concerns.

@Samoed

Samoed commented Feb 21, 2025

Copy link
Copy Markdown
Member Author

These tasks will still have type="Reranking" in their TaskMetadata, so this won’t break anything. However, for now, I think I’ll roll back the RerankingTask deletion since there might be more reranking tasks from main that need to be uploaded. I’ll wait for #2116 to be merged, as it includes a new reranking task.

@Samoed Samoed merged commit 8c27b80 into v2.0.0 Feb 24, 2025
@Samoed Samoed deleted the reupload_retrieval_datasets branch February 24, 2025 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@orionw orionw orionw approved these changes
@KennethEnevoldsen KennethEnevoldsen Awaiting requested review from KennethEnevoldsen

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Samoed @orionw