This is important for two reasons:
- If we stick to
torchdata for datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows.
- Our classification references have the ability to cache the dataset which includes the transformations:
|
utils.save_on_master((dataset, traindir), cache_path) |
So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.
cc @vfdev-5 @datumbox @bjuncek
This is important for two reasons:
torchdatafor datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows.vision/references/video_classification/train.py
Line 189 in 6e203b4
So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.
cc @vfdev-5 @datumbox @bjuncek