Skip to content

DNN: make MatMul support 3D or 4D with broadcast#22828

Merged
asmorkalov merged 1 commit intoopencv:4.xfrom
WanliZhong:improve_matmul
Dec 15, 2022
Merged

DNN: make MatMul support 3D or 4D with broadcast#22828
asmorkalov merged 1 commit intoopencv:4.xfrom
WanliZhong:improve_matmul

Conversation

@WanliZhong
Copy link
Copy Markdown
Member

@WanliZhong WanliZhong commented Nov 18, 2022 •
edited
Loading

Merge with extra: opencv/opencv_extra#1018

This PR follows the #22775

The main purpose of this PR is making MatMul support the broadcast that the second input has less dimention than the first one. And let the operation support SIMD and multi-thread. Beacuse it doesn't support 1D Mat, only support MatMul like

2x3x4 mul 4x5 -> 2x3x5
2x3x4x5 mul 3x5x6 -> 2x3x4x6
  • 2 const inputs: create a virtual layer for the first input
  • 1 const input with CPU (whether or not using broadcast): use the SIMD and multi-thread flow which for InnerProduct
  • 1 const input with CUDA: broadcast inputs will fallback to CPU. Inputs with the same shape will use the cuda.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

@WanliZhong WanliZhong mentioned this pull request Nov 18, 2022
Merged
@WanliZhong WanliZhong added category: dnn category: dnn (onnx) ONNX suport issues in DNN module labels Nov 18, 2022
@WanliZhong WanliZhong force-pushed the improve_matmul branch 2 times, most recently from e72deee to 05508ee Compare November 24, 2022 09:32
@WanliZhong WanliZhong force-pushed the improve_matmul branch 2 times, most recently from c731aaf to 34da3c0 Compare December 1, 2022 07:23
@WanliZhong WanliZhong requested review from rogday and zihaomu December 1, 2022 08:21
@WanliZhong WanliZhong marked this pull request as ready for review December 1, 2022 10:02
@WanliZhong
Copy link
Copy Markdown
Member Author

WanliZhong commented Dec 9, 2022 •
edited
Loading

Because the upstream has supported tranA and tranB, I will fix the confilcts.

@WanliZhong WanliZhong marked this pull request as draft December 9, 2022 07:48
@WanliZhong WanliZhong marked this pull request as ready for review December 9, 2022 09:27
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Dec 13, 2022

@zihaomu @rogday Your turn.

zihaomu
zihaomu approved these changes Dec 13, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@zihaomu zihaomu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution. LGTM! 👍

@alalek
Copy link
Copy Markdown
Member

alalek commented Dec 14, 2022

Merge branch '4.x' into improve_matmul
fix conflicts

Use git rebase instead of merge commits to have clear changes. GitHub has issues with handling PRs which includes merge commits.
PR should have 1 commit. This is stated in contribution guidelines.

@WanliZhong
Copy link
Copy Markdown
Member Author

WanliZhong commented Dec 14, 2022 via email

rogday
rogday approved these changes Dec 15, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@rogday rogday left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@asmorkalov asmorkalov added this to the 4.7.0 milestone Dec 15, 2022
@asmorkalov asmorkalov merged commit ac6fb17 into opencv:4.x Dec 15, 2022
alalek
alalek reviewed Dec 20, 2022
View reviewed changes
modules/dnn/test/test_onnx_importer.cpp
testONNXModels("matmul_4d_init");

testONNXModels("matmul_init_2");
testONNXModels("matmul_init_bcast");
Copy link
Copy Markdown
Member

@alalek alalek Dec 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is failed OpenCL FP16 test:

[ RUN      ] Test_ONNX_layers.MatMul_init/1, where GetParam() = OCV/OCL_FP16
[ INFO:0@189.433] global onnx_importer.cpp:822 populateNet DNN/ONNX: loading ONNX v8 model produced by 'matmul_2d_init'. Number of nodes = 1, initializers = 1, inputs = 2, outputs = 1
[ INFO:0@189.433] global onnx_importer.cpp:724 parseOperatorSet DNN/ONNX: ONNX opset version = 17
[ INFO:0@189.433] global onnx_importer.cpp:991 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node_output_0!output) from domain='ai.onnx'
[ INFO:0@189.434] global onnx_importer.cpp:822 populateNet DNN/ONNX: loading ONNX v8 model produced by 'matmul_3d_init'. Number of nodes = 1, initializers = 1, inputs = 2, outputs = 1
[ INFO:0@189.434] global onnx_importer.cpp:724 parseOperatorSet DNN/ONNX: ONNX opset version = 17
[ INFO:0@189.434] global onnx_importer.cpp:991 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node_output_0!output) from domain='ai.onnx'
[ INFO:0@189.434] global onnx_importer.cpp:822 populateNet DNN/ONNX: loading ONNX v8 model produced by 'matmul_4d_init'. Number of nodes = 1, initializers = 1, inputs = 2, outputs = 1
[ INFO:0@189.434] global onnx_importer.cpp:724 parseOperatorSet DNN/ONNX: ONNX opset version = 17
[ INFO:0@189.434] global onnx_importer.cpp:991 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node_output_0!output) from domain='ai.onnx'
[ INFO:0@189.434] global onnx_importer.cpp:822 populateNet DNN/ONNX: loading ONNX v8 model produced by 'matmul_init_2'. Number of nodes = 2, initializers = 2, inputs = 3, outputs = 1
[ INFO:0@189.434] global onnx_importer.cpp:724 parseOperatorSet DNN/ONNX: ONNX opset version = 17
[ INFO:0@189.434] global onnx_importer.cpp:991 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node_output_0!outputY) from domain='ai.onnx'
[ INFO:0@189.434] global onnx_importer.cpp:991 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [Add]:(onnx_node_output_0!output) from domain='ai.onnx'
[ INFO:0@189.435] global onnx_importer.cpp:822 populateNet DNN/ONNX: loading ONNX v8 model produced by 'matmul_init_bcast'. Number of nodes = 1, initializers = 1, inputs = 2, outputs = 1
[ INFO:0@189.435] global onnx_importer.cpp:724 parseOperatorSet DNN/ONNX: ONNX opset version = 17
[ INFO:0@189.435] global onnx_importer.cpp:991 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node_output_0!output) from domain='ai.onnx'
/build/precommit_opencl_linux/4.x/opencv/modules/dnn/test/test_common.impl.hpp:74: Failure
Expected: (normL1) <= (l1), actual: 1.22411 vs 0.004
  |ref| = 6.9979562759399414
/build/precommit_opencl_linux/4.x/opencv/modules/dnn/test/test_common.impl.hpp:77: Failure
Expected: (normInf) <= (lInf), actual: 6.99796 vs 0.02
  |ref| = 6.9979562759399414
[ INFO:0@189.435] global ts.cpp:850 testTearDown Memory_usage (OpenCL): 3960 (base=0  current=0)
[  FAILED  ] Test_ONNX_layers.MatMul_init/1, where GetParam() = OCV/OCL_FP16 (2 ms)

@alalek alalek mentioned this pull request Dec 20, 2022
Merged
@alalek alalek mentioned this pull request Jan 8, 2023
Merged
@WanliZhong WanliZhong mentioned this pull request Jan 19, 2023
Merged
6 tasks
@WanliZhong WanliZhong deleted the improve_matmul branch May 16, 2023 12:33
@dkurt dkurt mentioned this pull request Aug 3, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alalek alalek alalek left review comments

@zihaomu zihaomu zihaomu approved these changes

+1 more reviewer

@rogday rogday rogday approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

category: dnn (onnx) ONNX suport issues in DNN module category: dnn feature test

Projects

None yet

Milestone

4.7.0

Development

Successfully merging this pull request may close these issues.

5 participants

@WanliZhong @asmorkalov @alalek @zihaomu @rogday