Update OpenVINO init of new GEMM layer by dkurt · Pull Request #24309 · opencv/opencv

dkurt · 2023-09-21T12:59:38Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

CI validation:

Checklist:

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom

build_image:Custom=ubuntu-openvino-2021.4.2:20.04
Xbuild_image:Custom=ubuntu-openvino-2022.1.0:20.04

test_modules:Custom=dnn,python2,python3,java,gapi,video

buildworker:Custom=linux-4
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

asmorkalov

👍
Build with OpenVINO 2022.1.1.

asmorkalov · 2023-09-21T14:29:55Z

Locally failed tests:

[  PASSED  ] 9440 tests.
[  FAILED  ] 17 tests, listed below:
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/1, where GetParam() = OCV/OCL
[  FAILED  ] Test_Model.Keypoints_face/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_Model.TextRecognitionWithCTCPrefixBeamSearch/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Linear/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Multiplication/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Constant/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.LinearWithConstant/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.MatmulWithTwoInputs/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Gemm/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Gemm_bias/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Conformance_Gemm_all_attributes/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Conformance_Gemm_alpha/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Conformance_Gemm_beta/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_layers.Conformance_Gemm_default_matrix_bias/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_nets.LResNet100E_IR/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_ONNX_nets.Resnet34_kinetics/0, where GetParam() = NGRAPH/CPU

dkurt · 2023-09-21T14:36:51Z

@asmorkalov, thanks, I will fix it.

dkurt · 2023-09-21T16:37:37Z

Remaining failed test: https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100366

[  FAILED  ] 1 test, listed below:
[  FAILED  ] Test_ONNX_nets.Googlenet/0, where GetParam() = NGRAPH/CPU

perf:

[  FAILED  ] 6 tests, listed below:
[  FAILED  ] Gemm.innerproduct/0, where GetParam() = (A=[768, 768], B=[768, 768], C=[768], trans_a=0, trans_b=0, NGRAPH/CPU)
[  FAILED  ] Gemm.innerproduct/2, where GetParam() = (A=[1024, 1024], B=[1024, 1024], C=[1024], trans_a=0, trans_b=0, NGRAPH/CPU)
[  FAILED  ] Gemm.innerproduct/4, where GetParam() = (A=[50, 768], B=[768, 2304], trans_a=0, trans_b=0, NGRAPH/CPU)
[  FAILED  ] Gemm.innerproduct/6, where GetParam() = (A=[197, 768], B=[768, 2304], trans_a=0, trans_b=0, NGRAPH/CPU)
[  FAILED  ] Gemm.innerproduct/8, where GetParam() = (A=[50, 1024], B=[1024, 3072], trans_a=0, trans_b=0, NGRAPH/CPU)
[  FAILED  ] Gemm.innerproduct/10, where GetParam() = (A=[197, 1024], B=[1024, 3072], trans_a=0, trans_b=0, NGRAPH/CPU)

fengyuentau · 2023-09-22T02:26:20Z

modules/dnn/src/layers/gemm_layer.cpp

        if (have_bias && const_C) {
-            auto bias_node = std::make_shared<ngraph::op::Constant>(ngraph::element::f32,
-                                              ngraph::Shape{(size_t)blobs.back().size[1]}, blobs.back().data);
+            Mat bias = blobs.back();


For the GoogleNet test, I suspect a manual broadcast is needed for bias. I saw somewhere before that it seems openvino nodes cannot perform auto-broadcast.

Problem was in OpenVINO iterpretation of the batch dimension. Adding reshape from [2, 1, 1024] back to [2, 1024] solved it.

fengyuentau · 2023-09-22T02:28:35Z

As for the perf tests, maybe you need to change initialization to pass the check:

opencv/modules/dnn/src/net_openvino.cpp

Lines 272 to 282 in b51a78d

    
           if (ld.id == 0) 
        
           { 
        
               CV_Assert((netInputLayer->outNames.empty() && ld.outputBlobsWrappers.size() == 1) || 
        
                         (netInputLayer->outNames.size() == ld.outputBlobsWrappers.size())); 
        
               for (int i = 0; i < ld.outputBlobsWrappers.size(); ++i) 
        
               { 
        
                   std::string outputName = netInputLayer->outNames.empty() ? ld.name : netInputLayer->outNames[i]; 
        
                   outputName = ld.outputBlobsWrappers.size() > 1 ? (outputName + "." + std::to_string(i)) : outputName; 
        
                   ld.outputBlobsWrappers[i].dynamicCast<NgraphBackendWrapper>()->name = outputName; 
        
               } 
        
           }

fengyuentau · 2023-09-22T08:09:56Z

I guess it does not make a lot of senses to run both Gemm.gemm and Gemm.innerproduct on the ngraph backend? They are both basically calling the same openvino nodes. If so, keep only one is ok to save time running tests, e.g. Gemm.gemm.

dkurt · 2023-09-22T08:19:22Z

@fengyuentau, we can keep for compatibility. There are small differences in nGraph construction related to the reshapes and others before MatMul.

fengyuentau · 2023-09-22T08:21:14Z

ok, makes sense then.

asmorkalov · 2023-09-22T10:01:07Z

the last issue on my side:

[ RUN      ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.00276947 vs 0.002
First run  |ref| = 148.53863525390625
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.00216675 vs 0.002
Second run  |ref| = 148.02413940429688
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU (1371 ms)

asmorkalov · 2023-09-22T10:01:42Z

OpenVINO 2022.1.1 with old host (no AVX2). Not sure if the failure is related.

dkurt · 2023-09-22T10:30:15Z

@asmorkalov, that's right, model is from Torch frameworks and this PR is only related to ONNX models.

fengyuentau

👍 I guess failed tests in gapi does not relate to these changes, right?

dkurt · 2023-09-23T05:22:31Z

@fengyuentau , yes, the problem is old: #24049

asmorkalov · 2023-09-27T06:31:43Z

@dkurt Could you take a look on the BuildBot test failure:

[ RUN      ] Test_ONNX_layers.OpenAI_CLIP_head/0, where GetParam() = NGRAPH/CPU
[ INFO:0@88.629] global onnx_importer.cpp:828 populateNet DNN/ONNX: loading ONNX v8 model produced by 'clip-vit-base-head'. Number of nodes = 5, initializers = 5, inputs = 1, outputs = 1
[ INFO:0@88.629] global onnx_importer.cpp:729 parseOperatorSet DNN/ONNX: ONNX opset version = 18
[ INFO:0@88.629] global onnx_importer.cpp:1002 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [Mul]:(onnx_node_output_0!input_mul) from domain='ai.onnx'
[ INFO:0@88.629] global onnx_importer.cpp:1002 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [Expand]:(onnx_node_output_0!expand) from domain='ai.onnx'
[ INFO:0@88.629] global onnx_importer.cpp:1002 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node_output_0!concat) from domain='ai.onnx'
[ INFO:0@88.629] global onnx_importer.cpp:1002 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [Gather]:(onnx_node_output_0!gather) from domain='ai.onnx'
[ INFO:0@88.629] global onnx_importer.cpp:1002 handleNode DNN/ONNX: processing node with 2 inputs and 1 outputs: [Add]:(onnx_node_output_0!output) from domain='ai.onnx'
[ INFO:0@88.629] global net_openvino.cpp:705 switchToOpenVINOBackend DNN: switching to OpenVINO backend... (networkID=16678)
[ INFO:0@88.629] global ie_ngraph.cpp:766 initPlugin DNN/IE: Calling LoadNetwork(device=CPU)...
/build/precommit_custom_linux/4.x/opencv/modules/dnn/test/test_common.impl.hpp:76: Failure
Expected: (normL1) <= (l1), actual: 2.5 vs 1e-05
clip-vit-base-head  |ref| = 6.8675580024719238
/build/precommit_custom_linux/4.x/opencv/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 6 vs 0.0001
clip-vit-base-head  |ref| = 6.8675580024719238
[  FAILED  ] Test_ONNX_layers.OpenAI_CLIP_head/0, where GetParam() = NGRAPH/CPU (5 ms)

fengyuentau · 2023-09-27T07:02:31Z

Test_ONNX_layers.OpenAI_CLIP_head/0

Reference outputs of clip head is fixed via #24295. You can rebase opencv_extra and see whether it fixes the problem.

fengyuentau · 2023-09-27T07:05:50Z

DNN crash is being fixed in #24315

Update OpenVINO init of new GEMM layer opencv#24309 ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request CI validation: - [x] 2022.1.0: https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100368 - [ ] 2021.4.2: https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100373 Checklist: - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

Update OpenVINO init of new GEMM layer

fe902b1

dkurt requested a review from fengyuentau September 21, 2023 13:01

asmorkalov added this to the 4.9.0 milestone Sep 21, 2023

asmorkalov added the category: dnn label Sep 21, 2023

asmorkalov approved these changes Sep 21, 2023

View reviewed changes

Use alpha, beta parameters

e42e639

fengyuentau reviewed Sep 22, 2023

View reviewed changes

dkurt added 2 commits September 22, 2023 10:57

Workaround GoogleNet failure

a514665

Fix performance test

7cbb82d

asmorkalov self-assigned this Sep 22, 2023

fengyuentau approved these changes Sep 22, 2023

View reviewed changes

asmorkalov merged commit 2b6d0f3 into opencv:4.x Sep 27, 2023

dkurt deleted the gemm_ov_hotfix branch September 27, 2023 07:26

asmorkalov mentioned this pull request Sep 28, 2023

(5.x) Merge 4.x #24338

Merged

Uh oh!

Conversation

dkurt commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

asmorkalov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Sep 21, 2023

Uh oh!

dkurt commented Sep 21, 2023

Uh oh!

dkurt commented Sep 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengyuentau Sep 22, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Sep 22, 2023

Choose a reason for hiding this comment

Uh oh!

fengyuentau commented Sep 22, 2023

Uh oh!

fengyuentau commented Sep 22, 2023

Uh oh!

dkurt commented Sep 22, 2023

Uh oh!

fengyuentau commented Sep 22, 2023

Uh oh!

asmorkalov commented Sep 22, 2023

Uh oh!

asmorkalov commented Sep 22, 2023

Uh oh!

dkurt commented Sep 22, 2023

Uh oh!

fengyuentau left a comment

Choose a reason for hiding this comment

Uh oh!

dkurt commented Sep 23, 2023

Uh oh!

asmorkalov commented Sep 27, 2023

Uh oh!

fengyuentau commented Sep 27, 2023

Uh oh!

fengyuentau commented Sep 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dkurt commented Sep 21, 2023 •

edited

Loading

asmorkalov left a comment •

edited

Loading

dkurt commented Sep 21, 2023 •

edited

Loading