partial support for quantized models in ONNX importer by vpisarev · Pull Request #20264 · opencv/opencv

vpisarev · 2021-06-12T05:52:52Z

See #20188. Merge together with opencv/opencv_extra#878.

Note that the code is not fully tested. There is just a smoke test that checks that one simple quantized model is loaded successfully, and then we can run inference on it and get 'some' result of the proper type and size. We need some real quantized deep nets to test the functionality properly.
QLinearConv is converted to a normal Convolution layer; QLinearMatMul is converted to MatMul. The original 8-bit weights and the quantization/de-quantization factors are preserved though, so Convlution and MatMul layers can, in principle, be extended to support 8-bit weights.
QuantizeLinear outputs FP32 data; it needs to be extended to output 8-bit tensors.
DequantizeLinear and QLinearAdd take FP32, as well as INT8/UINT8 inputs, but the output is always FP32 for now.

The PR is submitted to master branch, rather than 3.4, because the whole support for 8-bit compute paths will be added to master (see #20228)

Pull Request Readiness Checklist

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom
build_image:Custom=ubuntu-openvino-2021.3.0:20.04
build_image:Custom Win=openvino-2021.3.0
build_image:Custom Mac=openvino-2021.3.0

test_modules:Custom=dnn,python2,python3,java
test_modules:Custom Win=dnn,python2,python3,java
test_modules:Custom Mac=dnn,python2,python3,java

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

…Linear, QLinearAdd to support 2D blobs as well; make sure the smoke test passes

…a file to opencv_extra/testdata/dnn/onnx/models subdirectory

alalek · 2021-06-15T09:24:05Z

modules/dnn/include/opencv2/dnn/all_layers.hpp

+        static Ptr<DequantizeLinearLayer> create(const LayerParams& params);
+    };
+
+    class CV_EXPORTS QLinearAddLayer : public Layer


It makes sense to add documentation with links on similar layers/nodes (e.g, from ONNX)

alalek · 2021-06-15T09:26:03Z

modules/dnn/src/layers/convolution_layer.cpp


        CV_Assert((inputs.size() > outputs.size() && blobs.empty()) ||
-                  (!inputs.empty() && (blobs.size() == 1 || blobs.size() == 2)));
+                  (!inputs.empty() && blobs.size() >= 1));


what this class should do with blobs.size() = 3 ?

This is assertion check about the code below.
It is caller responsibility to properly call this code without assertions

alalek · 2021-06-15T09:27:41Z

modules/dnn/src/layers/qlinearadd_layer.cpp

+// Copyright (C) 2016, Intel Corporation, all rights reserved.
+// Third party copyrights are property of their respective owners.


probably this is incorrect

alalek · 2021-06-15T09:31:30Z

modules/dnn/src/onnx/onnx_importer.cpp

+                int sc_total = (int)scale.total();
+                if (!scale.empty()) {
+                    if(!((sc_total == 1 || (k == 1 && sc_total == outN)) && scale.type() == CV_32F))
+                        CV_Error(CV_StsError,


CV_StsError

C API constants should not be used. Here and below.

alalek · 2021-06-15T09:39:04Z

modules/dnn/test/test_onnx_importer.cpp

+    ASSERT_FALSE(net.empty());
+    net.setPreferableBackend(backend);
+    net.setPreferableTarget(target);
+    std::cout << net.dump() << "\n";


if (cvtest::debugLevel > 0) or remove

alalek · 2021-06-15T09:39:54Z

modules/dnn/test/test_onnx_importer.cpp

+    net.setInput(input);
+    Mat output = net.forward();
+    //std::cout << output << std::endl;
+    ASSERT_EQ(output.size(), Size(128, 1));


*_EQ(expected_value, actual_value)

alalek · 2021-06-15T09:41:19Z

modules/dnn/test/test_onnx_importer.cpp

+TEST_P(Test_ONNX_nets, QuantStatic)
+{
+    if (backend != DNN_BACKEND_OPENCV || target != DNN_TARGET_CPU)
+        throw SkipTestException("Only the default backend; CPU target is supported");


default backend

default backend in builds with IE is IE, so this message should be corrected to be accurate.

asmorkalov · 2021-08-03T06:51:14Z

@vpisarev Friendly reminder.

vpisarev added 2 commits June 11, 2021 03:06

adding support for quantized ONNX nets into ONNX importer

fb478db

added QLinearAdd layer, made some fixes in QuantizeLinear, Dequantize…

ab0d523

…Linear, QLinearAdd to support 2D blobs as well; make sure the smoke test passes

vpisarev mentioned this pull request Jun 12, 2021

added very simple ONNX quantized model to test the support for such models in ONNX importer opencv/opencv_extra#878

Closed

fixed compile warnings on Linux, removed extra whitespaces, moved dat…

04327db

…a file to opencv_extra/testdata/dnn/onnx/models subdirectory

alalek reviewed Jun 15, 2021

View reviewed changes

jebastin-nadar mentioned this pull request Aug 11, 2021

dnn : int8 quantized layers support in onnx importer #20535

Merged

11 tasks

vpisarev closed this Aug 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

partial support for quantized models in ONNX importer#20264

partial support for quantized models in ONNX importer#20264
vpisarev wants to merge 3 commits intoopencv:masterfrom
vpisarev:onnx_q

vpisarev commented Jun 12, 2021 •

edited by alalek

Loading

Uh oh!

alalek Jun 15, 2021

Uh oh!

alalek Jun 15, 2021

Uh oh!

alalek Jun 15, 2021

Uh oh!

alalek Jun 15, 2021

Uh oh!

alalek Jun 15, 2021

Uh oh!

alalek Jun 15, 2021

Uh oh!

alalek Jun 15, 2021

Uh oh!

asmorkalov commented Aug 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// Copyright (C) 2016, Intel Corporation, all rights reserved.
		// Third party copyrights are property of their respective owners.

Uh oh!

Conversation

vpisarev commented Jun 12, 2021 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Aug 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vpisarev commented Jun 12, 2021 •

edited by alalek

Loading