The documentation is not available anymore as the PR was closed or merged.

Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424

The tests are failing due to the changed metadata:

got an unexpected keyword argument 'train-eval-index'

I think you can fix this by updating the DatasetMetadata class and implementing an appropriate validate_train_eval_index() function

@lhoestq we are working with an arbitrary set of tags for autoeval config. See https://github.com/huggingface/autonlp-backend/issues/414
I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this?

On the question of validating the train-eval-index metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).

For me, the required fields include:

• config
• task
• task_id
• splits (train / validation / eval)
• col_mapping
• metrics (checking that each one has type, name)

Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.

WDYT @lhoestq ?

Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it

I ended up improving the metadata validation in this PR x)

In particular:

• I added support YAML keys with dashes instead of underscores for train-eval-index
• I added train-eval-index validation with validate_train_eval_index. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want

Let me know if it sounds good to you ! I think we can improve validate_train_eval_index in another PR

Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x)

Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x)

Hehe, thanks!

Thanks, @lhoestq this is great!

Did I just fix it for windows and now it fails on linux ? xD

Did I just fix it for windows and now it fails on linux ? xD

Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅

The worst is that the tests pass locally both on my windows and my linux x)

Ok fixed it, the issue came from python 3.6 that doesn't return the right __origin__ for Dict and List types

Alright thanks for adding the first Autoeval config ! :D

Woohoo! Thank you so much 🤗

This is cool!