Skip to content

torchvision failures with NVFuser #76791

@davidberard98

Description

@davidberard98

🐛 Describe the bug

There are two errors thrown in torchvision tests when nvfuser is enabled:

first error

Error:

Traceback (most recent call last):
  File "/scratch/dberard/local/vision/test/test_models.py", line 768, in test_detection_model
    _check_jit_scriptable(model, ([x],), unwrapper=script_model_unwrapper.get(model_name, None), eager_out=out)
  File "/scratch/dberard/local/vision/test/test_models.py", line 140, in _check_jit_scriptable
    script_out = sm(*args)
  File "/scratch/dberard/local/pytorch/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: profiled_ints.size() == input_ints.size() && std::equal( profiled_ints.begin(), profiled_ints.end(), input_ints.begin()) INTERNAL ASSERT FAILED at "../torch/csrc/jit/codegen/cuda/parser.cpp":3045, please report a bug to PyTorch. profiling ivalue doesn't support merge

Repro: from the torchvision directory, run python -m pytest test/test_models.py -k cuda-fasterrcnn_mobilenet_v3_large_fpn.

There are also a number of other test_detection_model tests that are failing, which can be tested with python -m pytest test/test_models.py -k test_detection_model

second error

Error:

... (many similar errors) ...
________________________________________________________________________________________________________ TestColorJitter.test_color_jitter_all[3-cuda-9] ________________________________________________________________________________________________________
Traceback (most recent call last):
  File "/scratch/dberard/local/vision/test/test_transforms_tensor.py", line 186, in test_color_jitter_all
    _test_class_op(
  File "/scratch/dberard/local/vision/test/test_transforms_tensor.py", line 81, in _test_class_op
    transformed_tensor_script = scripted_fn(tensor)
  File "/scratch/dberard/local/pytorch/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
torch.jit.Error: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/scratch/dberard/local/vision/torchvision/transforms/functional_tensor.py", line 65, in forward
    c = get_dimensions(img)[0]
    if c not in permitted:
        raise TypeError(f"Input image tensor permitted channel values are {permitted}, but found {c}")
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
builtins.TypeError: Input image tensor permitted channel values are [1, 3], but found 26

======================================================================================================================= warnings summary ========================================================================================================================
torchvision/transforms/functional_pil.py:243
  /scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:243: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
    interpolation: int = Image.BILINEAR,

torchvision/transforms/functional_pil.py:317
  /scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:317: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
    interpolation: int = Image.NEAREST,

torchvision/transforms/functional_pil.py:333
  /scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:333: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
    interpolation: int = Image.NEAREST,

torchvision/transforms/functional_pil.py:350
  /scratch/dberard/local/vision/torchvision/transforms/functional_pil.py:350: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
    interpolation: int = Image.BICUBIC,

test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
  /scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %189 : int = prim::profile_ivalue(%187)
   does not have profile information (Triggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
    return forward_call(*input, **kwargs)

test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
  /scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %194 : int = prim::profile_ivalue(%dtype.1)
   does not have profile information (Triggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
    return forward_call(*input, **kwargs)

test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
  /scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %282 : int = prim::profile_ivalue(%280)
   does not have profile information (Triggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
    return forward_call(*input, **kwargs)

test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
  /scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %571 : int = prim::profile_ivalue(%orig_dtype.5)
   does not have profile information (Triggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
    return forward_call(*input, **kwargs)

test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[1-cpu-0]
  /scratch/dberard/local/pytorch/torch/nn/modules/module.py:1129: UserWarning: createConditionalConstant profile_node %561 : int = prim::profile_ivalue(%559)
   does not have profile information (Triggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:108.)
    return forward_call(*input, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
==================================================================================================================== short test summary info ====================================================================================================================
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-1] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-2] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-3] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-4] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-5] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-6] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-7] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-8] - torch.jit.Error: The following operation failed in the TorchScript interpreter.
FAILED test/test_transforms_tensor.py::TestColorJitter::test_color_jitter_all[3-cuda-9] - torch.jit.Error: The following operation failed in the TorchScript interpreter.

Repro: python -m pytest test/test_transforms_tensor.py -k test_color_jitter_all

Versions

torchvision - main branch
pytorch - #76006
A100

cc @fmassa @vfdev-5 @pmeier

Metadata

Metadata

Assignees

Labels

module: nvfusermodule: visiontriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions