Motivation

Our tests are getting a bit lengthy and considering we are adding new features, we are actually placing multiple tests in single file.
This leads to problems as the tests become harder to refactor and harder to read, manage. Standalone files for tests or grouped according to purpose are easier to anaylse and debug as well.

Our longest test files

Copy paste-able command

find test -name "*.py" | xargs -I {} wc -l {} | sort -nr | head -n 10

oke@psycon:~/Aditya/PyTorch/vision$ find test -name "*.py" | xargs -I {} wc -l {} | sort -nr | head -n 10
2675 test/test_datasets.py
2242 test/test_transforms.py
1535 test/builtin_dataset_mocks.py
1528 test/test_ops.py
1343 test/test_functional_tensor.py
1339 test/test_prototype_transforms_functional.py
1254 test/test_video_reader.py
982 test/test_transforms_tensor.py
954 test/datasets_utils.py
942 test/test_models.py

Many splits are possible that can make the tests smaller and easier to manage

E.g. 

test_models.py
-----  test_classification_models.py
----- test_detection_models.py
----- test_video_models.py

test_ops.py
------ test_post_processors.py
------ test_losses.py
------ test_layers.py

Possibilities are many, we can have multiple files with better grouping.

Pitch

We should think of grouping tests in logically similar way together. or consider creating test folder a package. See
#4436 . Earlier we have faced problems doing the same.

From discussion with Phillip offline
Considering we don't Pin pytest. We can make use of pytest-dev/pytest#9134

With the python path param, we can probably make use of common_utils.py file as well. So that we can test files without hurting other scenarios which were mentioned in the comments.

Alternatives

This is quite tricky but very helpful as we don't want tests to hurt us later.

Reference

#4436 (comment)

#4436 (comment)

#4436 (comment)

cc @NicolasHug @datumbox @pmeier