dnn: fix inconsistent input dtype for nary eltwise layers by fengyuentau · Pull Request #24386 · opencv/opencv

fengyuentau · 2023-10-11T03:49:58Z

Resolves #24385
Merge with opencv/opencv_extra#1107
Relates #24092 (comment)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

vpisarev · 2023-10-11T06:15:25Z

@opencv-alalek, the label "test" means what here? The new test was added that tests the case

modules/dnn/src/onnx/onnx_importer.cpp

fengyuentau · 2023-10-11T08:25:14Z

@dkurt Looks like a lot problems occur after the latest commit.

opencv-alalek · 2023-10-11T09:14:06Z

@opencv-alalek, the label "test" means what here? The new test was added that tests the case

There are 2 labels: "test" (PR includes appropriate test) / "pr: needs test" (test is required for new functionality / bugfix)

dkurt · 2023-10-11T09:29:47Z

@fengyuentau, there is one failed test [ FAILED ] DNNTestNetwork.CRNN/0, where GetParam() = OCV/CPU which refers to Concat layer:

[ RUN      ] DNNTestNetwork.CRNN/0, where GetParam() = OCV/CPU
[ERROR:0@32.375] global onnx_importer.cpp:1029 handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node!Concat_46) from domain='ai.onnx'
/home/ci/opencv/modules/ts/src/ts_perf.cpp:1965: Failure
Failed
Expected: PerfTestBody() doesn't throw an exception.
  Actual: it throws cv::Exception:
  OpenCV(4.8.0-dev) /home/ci/opencv/modules/dnn/src/onnx/onnx_importer.cpp:1051: error: (-2:Unspecified error) in function 'handleNode'
> Node [Concat@ai.onnx]:(onnx_node!Concat_46) parse error: OpenCV(4.8.0-dev) /home/ci/opencv/modules/dnn/src/onnx/onnx_importer.cpp:386: error: (-213:The function/feature is not implemented) Mixed input data types. in function 'runLayer'

fengyuentau · 2023-10-11T09:36:51Z

There are more in the "Accuracy: dnn" category:

[  FAILED  ] 5 tests, listed below:
[  FAILED  ] Test_Int8_layers.Reshape/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_ONNX_layers.Convolution_variable_weight_bias/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_ONNX_layers.DynamicReshape/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_ONNX_layers.CalculatePads/0, where GetParam() = OCV/CPU
[  FAILED  ] Test_ONNX_layers.Conv1d_variable_weight_bias/0, where GetParam() = OCV/CPU

dkurt · 2023-10-11T09:41:10Z

The reason is the same - "Mixed input data types". So can you please determine what actually data types are passed to Concat?

fengyuentau · 2023-10-11T14:40:55Z

Mixed input data types

The left two inputs of Concat is CV_32S, while the right input of Concat is CV_32F with this patch. Maybe we should do the type convertion in runtime instead of importing stage.

fengyuentau · 2023-10-11T15:56:43Z

Mixed input data types

The left two inputs of Concat is `CV_32S`, while the right input of Concat is `CV_32F` with this patch. Maybe we should do the type convertion in runtime instead of importing stage.

Update:

Runtime type convertion does not work. I did some type checks in const_layer.cpp with the new test case,

opencv/modules/dnn/src/layers/const_layer.cpp

Lines 81 to 83 in 590f150

    
           std::vector<Mat> outputs; 
        
           outputs_arr.getMatVector(outputs); 
        
           blobs[0].copyTo(outputs[0]);

tried to print the type of outputs[0] before and after blobs[0].copyTo, and found that outputs[0] did change to CV_32S after, but its type is somehow cast to CV_32F in the forward in nary_eltwise_layer.cpp.

Another update:

Trying a type conversion in const layer in runtime.

modules/dnn/src/layers/const_layer.cpp

dkurt

Thanks for the fix!

Merge with opencv/opencv#24386 Also noted that random seed is added in this pr, so previous data is regenerated to keep consistency.

dnn: fix inconsistent input dtype for nary eltwise layers opencv#24386 Resolves opencv#24385 Merge with opencv/opencv_extra#1107 Relates opencv#24092 (comment) ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

fix input dtype; add test

d7ac407

fengyuentau added bug category: dnn category: dnn (onnx) ONNX suport issues in DNN module labels Oct 11, 2023

fengyuentau added this to the 4.9.0 milestone Oct 11, 2023

fengyuentau requested a review from dkurt October 11, 2023 03:49

fengyuentau assigned vpisarev Oct 11, 2023

This was referenced Oct 11, 2023

add model and test data for fixing dtype bug in nary eltwise layers opencv/opencv_extra#1107

Merged

GSoC Add ONNX Support for GatherElements #24092

Merged

opencv-alalek added the test label Oct 11, 2023

dkurt reviewed Oct 11, 2023

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Oct 11, 2023

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

review comment

224c91d

fengyuentau and others added 6 commits October 12, 2023 00:00

try type convertion in runtime in const layer

6bb40a7

try replace copyTo with convertTo; fix cuda backend

1eb413e

try fix openvino

b172b23

fix typo

071e044

fix missing )

3bd3bea

try fix opencl

13aac66

dkurt reviewed Oct 12, 2023

View reviewed changes

modules/dnn/src/layers/const_layer.cpp Outdated Show resolved Hide resolved

asmorkalov requested a review from alexlyulkov October 12, 2023 08:18

fengyuentau changed the title ~~dnn: fix incorrect input dtype in nary eltwise layers~~ dnn: fix incorrect input dtype for nary eltwise layers Oct 12, 2023

try condition type != CV_32F

d45ebd9

dkurt approved these changes Oct 13, 2023

View reviewed changes

fengyuentau changed the title ~~dnn: fix incorrect input dtype for nary eltwise layers~~ dnn: fix inconsistent input dtype for nary eltwise layers Oct 13, 2023

asmorkalov merged commit 0507043 into opencv:4.x Oct 13, 2023

fengyuentau deleted the fix_dtype_nary_eltwise branch October 13, 2023 09:01

asmorkalov mentioned this pull request Oct 17, 2023

(5.x) Merge 4.x #24416

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

dnn: fix inconsistent input dtype for nary eltwise layers#24386

dnn: fix inconsistent input dtype for nary eltwise layers#24386
asmorkalov merged 9 commits intoopencv:4.xfrom
fengyuentau:fix_dtype_nary_eltwise

fengyuentau commented Oct 11, 2023

Uh oh!

vpisarev commented Oct 11, 2023

Uh oh!

Uh oh!

Uh oh!

fengyuentau commented Oct 11, 2023

Uh oh!

opencv-alalek commented Oct 11, 2023

Uh oh!

dkurt commented Oct 11, 2023 •

edited

Loading

Uh oh!

fengyuentau commented Oct 11, 2023

Uh oh!

dkurt commented Oct 11, 2023

Uh oh!

fengyuentau commented Oct 11, 2023

Uh oh!

fengyuentau commented Oct 11, 2023 •

edited

Loading

Uh oh!

Uh oh!

dkurt left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

fengyuentau commented Oct 11, 2023

Pull Request Readiness Checklist

Uh oh!

vpisarev commented Oct 11, 2023

Uh oh!

Uh oh!

Uh oh!

fengyuentau commented Oct 11, 2023

Uh oh!

opencv-alalek commented Oct 11, 2023

Uh oh!

dkurt commented Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengyuentau commented Oct 11, 2023

Uh oh!

dkurt commented Oct 11, 2023

Uh oh!

fengyuentau commented Oct 11, 2023

Uh oh!

fengyuentau commented Oct 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dkurt left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dkurt commented Oct 11, 2023 •

edited

Loading

fengyuentau commented Oct 11, 2023 •

edited

Loading