@orionw Can you please check this?

I've changed yor base branch to v2, because in your history was a lot of PRs from there.

I've changed yor base branch to v2, because in your history was a lot of PRs from there.

Thanks, I was just about to do that. Feel free to also give it a look. It is quite different from the previous version because v2 already has many of the implementations.

This pull request and #1947 are adding the same tasks? If so, maybe the old pull request should be closed?

Hi @Samoed

I've refactored the datasets. I preprocessed them externally, and now they look like the retrieval/reranking datasets on HF. Hence, I've deleted all the transforming/loading. I also fixed formatting like citations and other points you mentioned.

They should work now.

For some reason, I cannot test them locally. The first time I ran the tests, they failed because they expected corpus, queries, and qrels as configs instead of default, so I did that.

When I try to run tests now, it says it expects default as a config, so I am not sure.

I think issue with eval splits. In TaskMetadata you've annotated that they will use "test" split, but on huggingface you've uploaded train

I added the vscode folder.

There is only a test split for the dataset as far as I am aware because we are just evaluating.
It now shows the split as test on huggingface.

Can you please run it from your side and tell me if it works?

Yes, I'll try to run it. Can you resolve merge conflict?

Ah, yes. You should also calculate task.calculate_metadata_metrics() for each of the task

Do I add them in the classes for each task? And under which function?
Why isn't the calculation done in the abstract class, or is it just a one time thing?

You should call this function for each of the tasks. This function will produce json in descriptive_stats folder with info about dataset.

I am getting this error when I try to calculate metrics:
ValueError: BuilderConfig 'default' not found. Available: ['corpus', 'queries', 'qrels']

Isn't it expected to have corpus, queries, qrels as configs?

I tried changing qrels to default because apparently some datasets have it this way, but it is giving me this:
ValueError: Couldn't find cache for mteb/BIRCO-DorisMae-Test for config 'default'
Available configs in the cache: ['corpus', 'queries']

Can you please tell me what structure is the correct one?

I can download BIRCO-DorisMae without issues with 010fedd0de72e8b77388ea5b1d563d638d5900e5 revision

I can download BIRCO-DorisMae without issues with 010fedd0de72e8b77388ea5b1d563d638d5900e5 revision

I still haven't commited that revision. I've only made the change on HF. This revision has default as a config instead of qrels. Is this how it's supposed to be?

Yes, that dataset has right split format

Fixed License and now all datasets have default, queries, and corpus as configs.

Hi @Samoed

Another dataset seems to be causing the tests to fail "Namaa." Can you please confirm?

Yes, but that a bit weird, because I can find it on HF