🐛 Describe the bug
There are two errors thrown in torchvision tests when nvfuser is enabled:
first error
Error:
Traceback (most recent call last):
File "/scratch/dberard/local/vision/test/test_models.py", line 768, in test_detection_model
_check_jit_scriptable(model, ([x],), unwrapper=script_model_unwrapper.get(model_name, None), eager_out=out)
File "/scratch/dberard/local/vision/test/test_models.py", line 140, in _check_jit_scriptable
script_out = sm(*args)
File "/scratch/dberard/local/pytorch/torch/nn/modules/module.py", line 1129, in _call_impl
return forward_call(*input, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: profiled_ints.size() == input_ints.size() && std::equal( profiled_ints.begin(), profiled_ints.end(), input_ints.begin()) INTERNAL ASSERT FAILED at "../torch/csrc/jit/codegen/cuda/parser.cpp":3045, please report a bug to PyTorch. profiling ivalue doesn't support merge
Repro: from the torchvision directory, run python -m pytest test/test_models.py -k cuda-fasterrcnn_mobilenet_v3_large_fpn.
There are also a number of other test_detection_model tests that are failing, which can be tested with python -m pytest test/test_models.py -k test_detection_model
second error
Error:
... (many similar errors) ...
________________________________________________________________________________________________________ TestColorJitter.test_color_jitter_all[3-cuda-9] ________________________________________________________________________________________________________
Traceback (most recent call last):
File "/scratch/dberard/local/vision/test/test_transforms_tensor.py", line 186, in test_color_jitter_all
_test_class_op(
File "/scratch/dberard/local/vision/test/test_transforms_tensor.py", line 81, in _test_class_op
transformed_tensor_script = scripted_fn(tensor)
File "/scratch/dberard/local/pytorch/torch/nn/modules/module.py", line 1129, in _call_impl
return forward_call(*input, **kwargs)
torch.jit.Error: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "/scratch/dberard/local/vision/torchvision/transforms/functional_tensor.py", line 65, in forward
c = get_dimensions(img)[0]
if c not in permitted:
raise TypeError(f"Input image tensor permitted channel values are {permitted}, but found {c}")
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
builtins.TypeError: Input image tensor permitted channel values are [1, 3], but found 26
======================================================================================================================= warnings summary ========================================================================================================================
torchvision/transforms/functional_pil.py:243
/scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:243: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
interpolation: int = Image.BILINEAR,
torchvision/transforms/functional_pil.py:317
/scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:317: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
interpolation: int = Image.NEAREST,
torchvision/transforms/functional_pil.py:333
/scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:333: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
interpolation: int = Image.NEAREST,
torchvision/transforms/functional_pil.py:350
/scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:350: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
interpolation: int = Image.BICUBIC,
test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
/scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %189 : int = prim::profile_ivalue(%187)
does not have profile information (Triggered internally at ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
return forward_call(*input, **kwargs)
test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
/scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %194 : int = prim::profile_ivalue(%dtype.1)
does not have profile information (Triggered internally at ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
return forward_call(*input, **kwargs)
test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
/scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %282 : int = prim::profile_ivalue(%280)
does not have profile information (Triggered internally at ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
return forward_call(*input, **kwargs)
test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
/scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %571 : int = prim::profile_ivalue(%orig_dtype.5)
does not have profile information (Triggered internally at ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
return forward_call(*input, **kwargs)
test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
/scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %561 : int = prim::profile_ivalue(%559)
does not have profile information (Triggered internally at ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
return forward_call(*input, **kwargs)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================================================================================================== short test summary info ====================================================================================================================
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-1] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-2] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-3] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-4] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-5] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-6] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-7] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-8] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-9] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
Repro: python -m pytest test/test_transforms_tensor.py -k test_color_jitter_all
Versions
torchvision - main branch
pytorch - #76006
A100
cc @fmassa @vfdev-5 @pmeier
🐛 Describe the bug
There are two errors thrown in torchvision tests when nvfuser is enabled:
first error
Error:
Repro: from the torchvision directory, run
python -m pytest test/test_models.py -k cuda-fasterrcnn_mobilenet_v3_large_fpn.There are also a number of other
test_detection_modeltests that are failing, which can be tested withpython -m pytest test/test_models.py -k test_detection_modelsecond error
Error:
Repro:
python -m pytest test/test_transforms_tensor.py -k test_color_jitter_allVersions
torchvision - main branch
pytorch - #76006
A100
cc @fmassa @vfdev-5 @pmeier