Skip to content

dnn: add DepthToSpace and SpaceToDepth#25779

Merged
asmorkalov merged 13 commits intoopencv:4.xfrom
fengyuentau:dnn/fix_onnx_depthtospace
Jun 21, 2024
Merged

dnn: add DepthToSpace and SpaceToDepth#25779
asmorkalov merged 13 commits intoopencv:4.xfrom
fengyuentau:dnn/fix_onnx_depthtospace

Conversation

@fengyuentau
Copy link
Copy Markdown
Member

@fengyuentau fengyuentau commented Jun 18, 2024 •
edited
Loading

We are working on updating WeChat QRCode module. One of the new models is a fully convolutional model and hence it should be able to run with different input shapes. However, it has an operator DepthToSpace, which is parsed as a subgraph of Reshape -> Permute -> Reshape with a fixed shape getting during parsing. The subgraph itself is not a problem, but the true problem is the subgraph with a fixed input and output shape regardless input changes. This does not allow the model to run with different input shapes.

Solution is to add a dedicated layer for DepthtoSpace and SpaceToDepth.

Backend support:

  • CPU
  • CUDA
  • OpenCL
  • OpenVINO
  • CANN
  • TIMVX
  • Vulkan (missing fundamental tools, like permutation and reshape)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake
force_builders=Linux OpenCL,Win64 OpenCL

@asmorkalov asmorkalov added category: dnn (onnx) ONNX suport issues in DNN module feature labels Jun 18, 2024
@asmorkalov asmorkalov added this to the 4.11.0 milestone Jun 18, 2024
@fengyuentau

This comment was marked as resolved.

Sign in to view
@fengyuentau fengyuentau marked this pull request as ready for review June 20, 2024 10:08
@fengyuentau fengyuentau added category:dnn_timvx TIM-VX related issues in DNN module category:dnn_cann CANN backend related issues in DNN module labels Jun 20, 2024
@fengyuentau fengyuentau changed the title dnn: add dedicated DepthToSpace and SpaceToDepth layers dnn: add a dedicated DepthSpaceLayer for DepthToSpace and SpaceToDepth Jun 20, 2024
dkurt
dkurt reviewed Jun 20, 2024
View reviewed changes
dkurt
dkurt reviewed Jun 20, 2024
View reviewed changes
@fengyuentau fengyuentau changed the title dnn: add a dedicated DepthSpaceLayer for DepthToSpace and SpaceToDepth dnn: add DepthToSpace and SpaceToDepth Jun 20, 2024
zihaomu
zihaomu approved these changes Jun 21, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@zihaomu zihaomu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Jun 21, 2024

Failed CUDA tests:

[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA, where GetParam() = (test_depthtospace_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA, where GetParam() = (test_spacetodepth, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA, where GetParam() = (test_spacetodepth_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth_example, CUDA/CUDA_FP16)

@fengyuentau
Copy link
Copy Markdown
Member Author

fengyuentau commented Jun 21, 2024

Failed CUDA tests:

[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_crd_mode_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_crd_mode_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_dcr_mode_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_dcr_mode, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA, where GetParam() = (test_depthtospace_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_depthtospace_example_CUDA_CUDA_FP16, where GetParam() = (test_depthtospace_example, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA, where GetParam() = (test_spacetodepth, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth, CUDA/CUDA_FP16)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA, where GetParam() = (test_spacetodepth_example, CUDA/CUDA)
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_spacetodepth_example_CUDA_CUDA_FP16, where GetParam() = (test_spacetodepth_example, CUDA/CUDA_FP16)

Thank you for listing the error on CUDA backend. Hopefully the latest commit can fix this problem.

@asmorkalov asmorkalov merged commit 3f13ce7 into opencv:4.x Jun 21, 2024
@fengyuentau fengyuentau deleted the dnn/fix_onnx_depthtospace branch June 21, 2024 16:32
@asmorkalov asmorkalov mentioned this pull request Jul 1, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@asmorkalov asmorkalov asmorkalov left review comments

@zihaomu zihaomu zihaomu approved these changes

@dkurt dkurt dkurt approved these changes

Assignees

No one assigned

Labels

category:dnn_cann CANN backend related issues in DNN module category: dnn (onnx) ONNX suport issues in DNN module category:dnn_timvx TIM-VX related issues in DNN module feature

Projects

None yet

Milestone

4.11.0

Development

Successfully merging this pull request may close these issues.

4 participants

@fengyuentau @dkurt @asmorkalov @zihaomu