Is there an existing issue for this?

• I have searched the existing issues

Bug description

There are two ways to get the filenames for a model's dataset (.mat) and metadataset (.pickle) and mixing them can lead to incorrect evaluation results which can easily go undetected:

• The first way is to access the dataset and metadataset filepaths through the pose_cfg.yaml loaded as a dict eg. cfg["dataset"]. The only usages of this in the repository are to access the dataset filepath in the augmentation modules (/datasets directory). The metadataset (.pickle) file path is in fact never accessed through the pose_cfg.
• The second way is to use the auxiliary function get_data_and_metadata_filenames() which combines various variables to create the path from the provided args: trainingsetfolder, trainFraction, shuffle, cfg (config.yaml). This method is the one used the most often, mainly during the dataset creation but also notably during evaluation.

This means that if one wants to edit the pose_cfg.yaml of a model to use a specific training dataset other than the default one for that shuffle, it won't behave as intended. The augmentation modules (/datasets folder) use cfg["dataset"] to obtain the path to the .mat file and thus the training is done using the dataset defined in the configuration. However, the evaluation (evaluate.py) accesses this file path using get_data_and_metadata_filenames() and therefore uses the original, default dataset regardless of what dataset is specified in either the train/pose_cfg.yaml or test/pose_cfg.yaml.

For context, I am trying to use the modelprefix parameter to give a clear naming scheme to the groups of models I would like to train. In this workflow the training datasets I want to use don't need to correspond to any specific 'shuffle'.

Operating System

MacOS 12.3.1
(& Colab)

DeepLabCut version

dlc version 2.3.5

DeepLabCut mode

single animal

Device type

CPU

Steps To Reproduce

I can provide a script to demonstrate that that evaluation ignores the pose_cfg if required but it doesn't explicitly show that the training does use the parameter from the pose_cfg.

Relevant log output

No response

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct