This issue tracks work related to improving PyTorch's test suite.

Deprecations

• Remove assertEqualIgnoreTypes (see Remove all self.assertEqualIgnoreTypes instances #38095)

Test Suites needing updates

• test_binary_ufuncs.py
  • needs structured "forward testing" analogous to test_unary_ufuncs.py
  • needs automated type promotion testing
• test_reductions.py
  • needs structured "forward testing" analogous to test_unary_ufuncs.py (Write a ReductionInfo subclass of OpInfo #49746)

OpInfo-related improvements

• Fix ops failing test_out

General Framework Improvements

• Re-run failing CUDA tests with CUDA launch blocking (Automatically rerun tests with CUDA_LAUNCH_BLOCKING=1 when they fail with CUDA errors in CI #49023)
• Add an option to run only tests for a specific device type ([Feature Request] Add an option to run GPU tests only, and skip all CPU tests #44159)
• Debug and fix test suite memory issues (Some largeCUDATensorTest fails with OOM when running with the entire test suit, but not when running standalone #43677)

NN Module Testing

• Add "ModuleInfos" analogous to OpInfos to facilitate systemic testing
• Refactor test_nn.py to be fewer than 10k lines (Get test_jit.py below 10k lines #42847)
• Reduce the time taken by test_ReplicationPad3d (test_ReplicationPad3d (test_nn.TestNN) takes too long to run #38190)
• Reduce the time taken by test_LocalResponseNorm_3d_custom_params (test_LocalResponseNorm_3d_custom_params (test_nn.TestNN) takes too long to run #38189)
• Reduce the time taken by test_interpolate_nearest_scale_3d (test_interpolate_nearest_scale_3d in test_nn takes too long to run #38188)
• Review convolution testing (Test suite should test implementations #6135)

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @mruberry @VitalyFedyunin @walterddr