python3 ./tools/deploy.py configs/mmrotate/rotated-detection_tensorrt-fp16_dynamic-320x320-1024x1024.py /mmrotate/configs/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90.py /mmrotate/gliding_vertex_r50_fpn_1x_dota_le90-12e7423c.pth demo/resources/original.png --work-dir work-dir/ --show --device cuda

root@donggeun-ardet-debug--1-jtb6q:/mmdeploy# python3 ./tools/deploy.py configs/mmrotate/rotated-detection_tensorrt-fp16_dynamic-320x320-1024x1024.py /mmrotate/configs/gliding_vertex/gliding_vertex_r50_fpn_1x_dota_le90.py /mmrotate/gliding_vertex_r50_fpn_1x_dota_le90-12e7423c.pth demo/resources/original.png --work-dir work-dir/ --show --device cuda
/usr/local/lib/python3.8/site-packages/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/usr/local/lib/python3.8/site-packages/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
DEPLOY_CFG_PATH configs/mmrotate/rotated-detection_tensorrt-fp16_dynamic-320x320-1024x1024.py
AAAAAAAAAAAAAAAAA
Config (path: configs/mmrotate/rotated-detection_tensorrt-fp16_dynamic-320x320-1024x1024.py): {'onnx_config': {'type': 'onnx', 'export_params': True, 'keep_initializers_as_inputs': False, 'opset_version': 11, 'save_file': 'end2end.onnx', 'input_names': ['input'], 'output_names': ['dets', 'labels'], 'input_shape': None, 'dynamic_axes': {'input': {0: 'batch', 2: 'height', 3: 'width'}, 'dets': {0: 'batch', 1: 'num_dets'}, 'labels': {0: 'batch', 1: 'num_dets'}}}, 'codebase_config': {'type': 'mmrotate', 'task': 'RotatedDetection', 'post_processing': {'score_threshold': 0.05, 'iou_threshold': 0.1, 'pre_top_k': 3000, 'keep_top_k': 2000}}, 'backend_config': {'type': 'tensorrt', 'common_config': {'fp16_mode': True, 'max_workspace_size': 1073741824}, 'model_inputs': [{'input_shapes': {'input': {'min_shape': [1, 3, 320, 320], 'opt_shape': [1, 3, 1024, 1024], 'max_shape': [1, 3, 1024, 1024]}}}]}}
IR.ONNX
/usr/local/lib/python3.8/site-packages/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
2022-06-24 08:31:33,478 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
CODEBASE_TYPE Codebase.MMROTATE
/usr/local/lib/python3.8/site-packages/mmdet/models/dense_heads/anchor_head.py:116: UserWarning: DeprecationWarning: `num_anchors` is deprecated, for consistency or also use `num_base_priors` instead
  warnings.warn('DeprecationWarning: `num_anchors` is deprecated, '
SCALE_FACTOR
2022-06-24 08:31:38,208 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
2022-06-24 08:31:38,208 - mmdeploy - INFO - Export PyTorch model to ONNX: work-dir/end2end.onnx.
/mmdeploy/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/usr/local/lib/python3.8/site-packages/mmdet/models/dense_heads/anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead
  warnings.warn('DeprecationWarning: anchor_generator is deprecated, '
/usr/local/lib/python3.8/site-packages/mmdet/core/anchor/anchor_generator.py:333: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors`` 
  warnings.warn('``grid_anchors`` would be deprecated soon. '
/usr/local/lib/python3.8/site-packages/mmdet/core/anchor/anchor_generator.py:369: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors`` 
  warnings.warn(
/mmdeploy/mmdeploy/codebase/mmrotate/models/rotated_rpn_head.py:79: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]
torch.Size([1, 178176, 1]) torch.Size([1, 118784, 6]) torch.Size([1, 178176, 4])
2000
torch.Size([1, 180176, 1]) tensor([[[0.4740],
         [0.4997],
         [0.4976],
         ...,
         [0.0000],
         [0.0000],
         [0.0000]]], device='cuda:0')
/mmdeploy/mmdeploy/pytorch/functions/topk.py:56: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if k > size:
BBOX_PRED torch.Size([1, 120784, 6])
batch_inds tensor([[0]], device='cuda:0')
topk_inds tensor([[ 56699, 156938, 120050,  ..., 177839, 176799, 177824]],
       device='cuda:0')
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [56,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [58,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [59,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [60,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [8,0,0], thread: [61,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [6,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [12,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [12,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 105, in __call__
    ret = func(*args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 86, in torch2onnx
    export(
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 354, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 324, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 273, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 105, in __call__
    ret = func(*args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/onnx/export.py", line 121, in export
    torch.onnx.export(
  File "/usr/local/lib/python3.8/site-packages/torch/onnx/__init__.py", line 275, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/usr/local/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/usr/local/lib/python3.8/site-packages/torch/onnx/utils.py", line 689, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper
    return self.func(self, *args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/onnx/optimizer.py", line 10, in model_to_graph__custom_optimizer
    graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/onnx/utils.py", line 458, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args,
  File "/usr/local/lib/python3.8/site-packages/torch/onnx/utils.py", line 422, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/usr/local/lib/python3.8/site-packages/torch/onnx/utils.py", line 373, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/usr/local/lib/python3.8/site-packages/torch/jit/_trace.py", line 1160, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/usr/local/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper
    return self.func(self, *args, **kwargs)
  File "/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py", line 69, in base_detector__forward
    return __forward_impl(ctx, self, img, img_metas=img_metas, **kwargs)
  File "/mmdeploy/mmdeploy/core/optimizers/function_marker.py", line 261, in g
    rets = f(*args, **kwargs)
  File "/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py", line 28, in __forward_impl
    return self.simple_test(img, img_metas, **kwargs)
  File "/mmrotate/mmrotate/models/detectors/two_stage.py", line 179, in simple_test
    proposal_list = self.rpn_head.simple_test_rpn(x, img_metas)
  File "/usr/local/lib/python3.8/site-packages/mmdet/models/dense_heads/dense_test_mixins.py", line 130, in simple_test_rpn
    proposal_list = self.get_bboxes(*rpn_outs, img_metas=img_metas)
  File "/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper
    return self.func(self, *args, **kwargs)
  File "/mmdeploy/mmdeploy/codebase/mmrotate/models/rotated_rpn_head.py", line 117, in rotated_rpn_head__get_bboxes
    bbox_pred = bbox_pred[batch_inds, topk_inds, :]
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:1055 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f97b3d01a22 in /usr/local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x10aa3 (0x7f97b3f62aa3 in /usr/local/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a7 (0x7f97b3f64147 in /usr/local/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7f97b3ceb5a4 in /usr/local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0xa2822a (0x7f9797cbd22a in /usr/local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xa282c1 (0x7f9797cbd2c1 in /usr/local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0xdd680 (0x5641b5f78680 in /usr/local/bin/python3)
frame #7: <unknown function> + 0x5dca7 (0x5641b5ef8ca7 in /usr/local/bin/python3)
frame #8: <unknown function> + 0x19cd95 (0x5641b6037d95 in /usr/local/bin/python3)
frame #9: <unknown function> + 0x19cda5 (0x5641b6037da5 in /usr/local/bin/python3)
frame #10: <unknown function> + 0x19cda5 (0x5641b6037da5 in /usr/local/bin/python3)
frame #11: <unknown function> + 0x19cda5 (0x5641b6037da5 in /usr/local/bin/python3)
frame #12: _PyEval_EvalFrameDefault + 0x12dc (0x5641b5efb04c in /usr/local/bin/python3)
frame #13: _PyEval_EvalCodeWithName + 0xb60 (0x5641b5fd66b0 in /usr/local/bin/python3)
frame #14: _PyFunction_Vectorcall + 0x92 (0x5641b5f12252 in /usr/local/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5762 (0x5641b5eff4d2 in /usr/local/bin/python3)
frame #16: <unknown function> + 0x5db77 (0x5641b5ef8b77 in /usr/local/bin/python3)
frame #17: _PyEval_EvalFrameDefault + 0x5829 (0x5641b5eff599 in /usr/local/bin/python3)
frame #18: _PyEval_EvalCodeWithName + 0xb60 (0x5641b5fd66b0 in /usr/local/bin/python3)
frame #19: _PyFunction_Vectorcall + 0x92 (0x5641b5f12252 in /usr/local/bin/python3)
frame #20: _PyEval_EvalFrameDefault + 0x5718 (0x5641b5eff488 in /usr/local/bin/python3)
frame #21: _PyEval_EvalCodeWithName + 0xb60 (0x5641b5fd66b0 in /usr/local/bin/python3)
frame #22: PyEval_EvalCode + 0x23 (0x5641b5fd6753 in /usr/local/bin/python3)
frame #23: <unknown function> + 0x183c11 (0x5641b601ec11 in /usr/local/bin/python3)
frame #24: PyRun_SimpleStringFlags + 0x15b (0x5641b601f85b in /usr/local/bin/python3)
frame #25: Py_RunMain + 0x2d4 (0x5641b5f03eb4 in /usr/local/bin/python3)
frame #26: Py_BytesMain + 0x56 (0x5641b5f048c6 in /usr/local/bin/python3)
frame #27: __libc_start_main + 0xe7 (0x7f97cb9f1c87 in /lib/x86_64-linux-gnu/libc.so.6)
frame #28: _start + 0x2a (0x5641b5f0357a in /usr/local/bin/python3)

Traceback (most recent call last):
  File "./tools/deploy.py", line 369, in <module>
    main()
  File "./tools/deploy.py", line 134, in main
    torch2ir(ir_type)(
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 354, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 322, in call_function
    return self.get_result_sync(call_id)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 303, in get_result_sync
    ret = self.get_caller(func_name).pop_mp_output(call_id)
  File "/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 79, in pop_mp_output
    assert call_id in self._mp_dict, \
AssertionError: `mmdeploy.apis.pytorch2onnx.torch2onnx` with Call id: 0 failed.