TFLite models on different backends (tests and improvements) by dkurt · Pull Request #24039 · opencv/opencv

dkurt · 2023-07-21T10:44:58Z

Pull Request Readiness Checklist

MaxUnpooling with OpenVINO
Fully connected with transposed inputs/weights with OpenVINO
Enable backends tests for TFLite (related to Issue loading TFLite net #23992 (comment))
Increase existing tests thresholds

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom

Xbuild_image:Custom=ubuntu-openvino-2021.4.2:20.04
build_image:Custom=ubuntu-openvino-2022.1.0:20.04

test_modules:Custom=dnn,python2,python3,java,gapi,video

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=ON
test_bigdata:Custom=1
test_filter:Custom=*

dkurt · 2023-07-21T11:52:23Z

@asmorkalov, do we still have an option to run CI with OpenVINO setup somehow?

opencv-alalek · 2023-07-22T05:53:49Z

option to run CI with OpenVINO setup

Legacy CI:

weekly: https://pullrequest.opencv.org/buildbot/builders/4_x_openvino-skl-lin64 (openvino-2021.4.2)
CI configuration for PR has been added

asmorkalov · 2023-07-25T09:14:03Z

I see the following failure with the patch:

./bin/opencv_test_dnn --gtest_filter="DNNTestNetwork.FastNeuralStyle_eccv16*" 
...
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from DNNTestNetwork
[ RUN      ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU
/mnt/projects/Projects/OpenCV/opencv-master/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.00276947 vs 0.002
First run  |ref| = 148.53863525390625
/mnt/projects/Projects/OpenCV/opencv-master/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.00216675 vs 0.002
Second run  |ref| = 148.02413940429688
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU (1346 ms)
[ RUN      ] DNNTestNetwork.FastNeuralStyle_eccv16/1, where GetParam() = OCV/OCL
[ WARN:0@1.866] global ocl4dnn_conv_spatial.cpp:1923 loadTunedConfig OpenCV(ocl4dnn): consider to specify kernel configuration cache directory through OPENCV_OCL4DNN_CONFIG_PATH parameter.
Could not open file: /tmp/dep-02b26e.d
OpenCL program build log: dnn/dummy
Status -11: CL_BUILD_PROGRAM_FAILURE
-cl-no-subgroup-ifp
Error in processing command line: Don't understand command line argument "-cl-no-subgroup-ifp"!
/mnt/projects/Projects/OpenCV/opencv-master/modules/dnn/test/test_common.impl.hpp:79: Failure
Expected: (normInf) <= (lInf), actual: 0.00235748 vs 0.002
Second run  |ref| = 148.02413940429688
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/1, where GetParam() = OCV/OCL (1065 ms)
[ RUN      ] DNNTestNetwork.FastNeuralStyle_eccv16/2, where GetParam() = OCV/OCL_FP16
[       OK ] DNNTestNetwork.FastNeuralStyle_eccv16/2 (707 ms)
[----------] 3 tests from DNNTestNetwork (3118 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (3118 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/1, where GetParam() = OCV/OCL

Not sure if it's relevant. Will check 4.x and return back with details.

asmorkalov · 2023-07-25T09:38:14Z

The issue is not relevant to the PR and exists in 4.x too. OpenVINO version: openvino_toolkit_ubuntu18_2022.1.1.7030.39aba80957e

modules/dnn/src/layers/max_unpooling_layer.cpp

dkurt · 2023-07-28T11:39:27Z

[  PASSED  ] 13143 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] Test_Caffe_nets.FasterRCNN_vgg16/2, where GetParam() = NGRAPH/OCL_FP16
[  FAILED  ] Test_Darknet_nets.YOLOv4/2, where GetParam() = NGRAPH/OCL_FP16
[  FAILED  ] Test_ONNX_layers.LSTM_init_h0_c0/2, where GetParam() = NGRAPH/OCL_FP16

Number of HETERO:GPU,CPU fallbacks: 8500 (fallbacks to OpenCV implementation as custom layer)

asmorkalov · 2023-08-02T11:40:47Z

@dkurt could you rebase the patch and fix conflicts.

dkurt · 2023-08-02T11:50:11Z

@asmorkalov, I also waiting for https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100310 but it runs too long on the latest commit (previous were fine). So need to check that problem is not in the changes from the latest commit.

asmorkalov · 2023-08-03T06:55:54Z

modules/dnn/test/test_tflite_importer.cpp

    std::vector<std::vector<Mat> > outs;
-    net.forward(outs, {"p_re_lu_1", "max_pooling_with_argmax2d", "conv2d_86", "max_unpooling2d_2"});
+    if (backend == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH) {
+        // TODO: seems like a bug with a retrieving intermediate tensors


Please file a ticket about it.

asmorkalov · 2023-08-03T07:18:54Z

@dkurt Thanks a for the branch rebase. Looks like there is some interference with previous patches. I see a lot of new regressions with OpenVINO. Full list of failed test:

[  FAILED  ] 33 tests, listed below:
[  FAILED  ] DNNTestNetwork.AlexNet/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] DNNTestNetwork.GoogLeNet/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] DNNTestNetwork.FastNeuralStyle_eccv16/1, where GetParam() = OCV/OCL
[  FAILED  ] Test_Caffe_nets.FasterRCNN_vgg16/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_Caffe_nets.FasterRCNN_zf/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_Darknet_layers.connected/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/0, where GetParam() = (3, 5x4, 3, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/3, where GetParam() = (3, 5x4, 3, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/6, where GetParam() = (3, 5x4, 4, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/9, where GetParam() = (3, 5x4, 4, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/12, where GetParam() = (3, 4x5, 3, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/15, where GetParam() = (3, 4x5, 3, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/18, where GetParam() = (3, 4x5, 4, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/21, where GetParam() = (3, 4x5, 4, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/24, where GetParam() = (3, 1x1, 3, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/27, where GetParam() = (3, 1x1, 3, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/30, where GetParam() = (3, 1x1, 4, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/33, where GetParam() = (3, 1x1, 4, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/36, where GetParam() = (4, 5x4, 3, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/39, where GetParam() = (4, 5x4, 3, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/42, where GetParam() = (4, 5x4, 4, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/45, where GetParam() = (4, 5x4, 4, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/48, where GetParam() = (4, 4x5, 3, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/51, where GetParam() = (4, 4x5, 3, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/54, where GetParam() = (4, 4x5, 4, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/57, where GetParam() = (4, 4x5, 4, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/60, where GetParam() = (4, 1x1, 3, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/63, where GetParam() = (4, 1x1, 3, true, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/66, where GetParam() = (4, 1x1, 4, false, NGRAPH/CPU)
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/69, where GetParam() = (4, 1x1, 4, true, NGRAPH/CPU)
[  FAILED  ] Test_Caffe_layers.InnerProduct/0, where GetParam() = NGRAPH/CPU
[  FAILED  ] Test_Caffe_layers.layer_prelu_fc/0, where GetParam() = NGRAPH/CPU

Detailed logs:

[ RUN      ] DNNTestNetwork.AlexNet/0, where GetParam() = NGRAPH/CPU
unknown file: Failure
C++ exception with description "Check 'DimType::merge(merged_dimension, arg0_col_dim, arg1_row_dim) || arg0_col_dim.is_dynamic() || arg1_row_dim.is_dynamic()' failed at core/shape_inference/include/matmul_shape_inference.hpp:77:
While validating node 'v0::MatMul MatMul_34 (pool5[0]:f32{1,256,6,6}, Constant_33[0]:f32{1,1,4096,9216}) -> (dynamic...)' with friendly_name 'MatMul_34':
Incompatible MatMul matrix dimension. First input dimension=6 at COL_INDEX_DIM=3 doesn't match the second input dimension=9216 at ROW_INDEX_DIM=2
" thrown in the test body.
[  FAILED  ] DNNTestNetwork.AlexNet/0, where GetParam() = NGRAPH/CPU (1114 ms)

[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/0, where GetParam() = (3, 5x4, 3, false, NGRAPH/CPU)
unknown file: Failure
C++ exception with description "Check 'DimType::merge(merged_dimension, arg0_col_dim, arg1_row_dim) || arg0_col_dim.is_dynamic() || arg1_row_dim.is_dynamic()' failed at core/shape_inference/include/matmul_shape_inference.hpp:77:
While validating node 'v0::MatMul MatMul_2 (_input[0]:f32{1,3,4,5}, Constant_1[0]:f32{3,60}) -> (dynamic...)' with friendly_name 'MatMul_2':
Incompatible MatMul matrix dimension. First input dimension=5 at COL_INDEX_DIM=3 doesn't match the second input dimension=60 at ROW_INDEX_DIM=0
" thrown in the test body.
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/0, where GetParam() = (3, 5x4, 3, false, NGRAPH/CPU) (9 ms)
[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/1, where GetParam() = (3, 5x4, 3, false, OCV/OCL)
[       OK ] Layer_Test_Halide/FullyConnected.Accuracy/1 (2 ms)
[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/2, where GetParam() = (3, 5x4, 3, false, OCV/OCL_FP16)
[       OK ] Layer_Test_Halide/FullyConnected.Accuracy/2 (1 ms)
[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/3, where GetParam() = (3, 5x4, 3, true, NGRAPH/CPU)
unknown file: Failure
C++ exception with description "Check 'DimType::merge(merged_dimension, arg0_col_dim, arg1_row_dim) || arg0_col_dim.is_dynamic() || arg1_row_dim.is_dynamic()' failed at core/shape_inference/include/matmul_shape_inference.hpp:77:
While validating node 'v0::MatMul MatMul_5 (_input[0]:f32{1,3,4,5}, Constant_4[0]:f32{3,60}) -> (dynamic...)' with friendly_name 'MatMul_5':
Incompatible MatMul matrix dimension. First input dimension=5 at COL_INDEX_DIM=3 doesn't match the second input dimension=60 at ROW_INDEX_DIM=0
" thrown in the test body.
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/3, where GetParam() = (3, 5x4, 3, true, NGRAPH/CPU) (0 ms)
[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/4, where GetParam() = (3, 5x4, 3, true, OCV/OCL)
[       OK ] Layer_Test_Halide/FullyConnected.Accuracy/4 (0 ms)
[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/5, where GetParam() = (3, 5x4, 3, true, OCV/OCL_FP16)
[       OK ] Layer_Test_Halide/FullyConnected.Accuracy/5 (1 ms)
[ RUN      ] Layer_Test_Halide/FullyConnected.Accuracy/6, where GetParam() = (3, 5x4, 4, false, NGRAPH/CPU)
unknown file: Failure
C++ exception with description "Check 'DimType::merge(merged_dimension, arg0_col_dim, arg1_row_dim) || arg0_col_dim.is_dynamic() || arg1_row_dim.is_dynamic()' failed at core/shape_inference/include/matmul_shape_inference.hpp:77:
While validating node 'v0::MatMul MatMul_8 (_input[0]:f32{1,3,4,5}, Constant_7[0]:f32{4,60}) -> (dynamic...)' with friendly_name 'MatMul_8':
Incompatible MatMul matrix dimension. First input dimension=5 at COL_INDEX_DIM=3 doesn't match the second input dimension=60 at ROW_INDEX_DIM=0
" thrown in the test body.
[  FAILED  ] Layer_Test_Halide/FullyConnected.Accuracy/6, where GetParam() = (3, 5x4, 4, false, NGRAPH/CPU) (0 ms)

[ RUN      ] Test_Caffe_layers.InnerProduct/0, where GetParam() = NGRAPH/CPU
unknown file: Failure
C++ exception with description "Check 'DimType::merge(merged_dimension, arg0_col_dim, arg1_row_dim) || arg0_col_dim.is_dynamic() || arg1_row_dim.is_dynamic()' failed at core/shape_inference/include/matmul_shape_inference.hpp:77:
While validating node 'v0::MatMul MatMul_2 (input[0]:f32{2,6,75,113}, Constant_1[0]:f32{10,8475}) -> (dynamic...)' with friendly_name 'MatMul_2':
Incompatible MatMul matrix dimension. First input dimension=113 at COL_INDEX_DIM=3 doesn't match the second input dimension=8475 at ROW_INDEX_DIM=0
" thrown in the test body.
[  FAILED  ] Test_Caffe_layers.InnerProduct/0, where GetParam() = NGRAPH/CPU (10 ms)

dkurt · 2023-08-03T07:55:56Z

@asmorkalov, thanks! I will fix it.

dkurt · 2023-08-03T12:40:09Z

modules/dnn/test/test_halide_layers.cpp

-#endif
+    // https://github.com/openvinotoolkit/openvino/issues/18731
+    if (backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH && stride != Size(1, 1))
+        applyTestTag(CV_TEST_TAG_DNN_SKIP_IE_NGRAPH);


@asmorkalov, please do not merge, I'm going to workaround it as suggested in the ticket.

Please ignore, this must complicate this PR too much. Let's just skip for now

dkurt · 2023-08-04T06:07:15Z

@asmorkalov, I also waiting for https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100310 but it runs too long on the latest commit (previous were fine). So need to check that problem is not in the changes from the latest commit.

@asmorkalov, seems like Custom worker got stuck. The job is always "queued".

asmorkalov · 2023-08-04T07:53:57Z

The last version passes all tests with l_openvino_toolkit_ubuntu18_2022.1.1!

asmorkalov

👍

opencv-alalek · 2023-08-22T19:35:20Z

modules/dnn/src/layers/pooling_layer.cpp

+            std::shared_ptr<ngraph::Node> max_pool;
+            if (computeMaxIdx) {
+                std::vector<size_t> dilations(kernel_size.size(), 1);
+                max_pool = std::make_shared<ngraph::op::v8::MaxPool>(ieInpNode, ngraph::Strides(strides), ngraph::Strides(dilations),


There are broken weekly builds:

/build/4_x_openvino-opencl-skl-lin64/opencv/modules/dnn/src/layers/pooling_layer.cpp: In member function 'virtual cv::Ptr<cv::dnn::dnn4_v20230620::BackendNode> cv::dnn::PoolingLayerImpl::initNgraph(const std::vector<cv::Ptr<cv::dnn::dnn4_v20230620::BackendWrapper> >&, const std::vector<cv::Ptr<cv::dnn::dnn4_v20230620::BackendNode> >&)': /build/4_x_openvino-opencl-skl-lin64/opencv/modules/dnn/src/layers/pooling_layer.cpp:619:57: error: 'v8' is not a member of 'ngraph::op'; did you mean 'v7'? 619 | max_pool = std::make_shared<ngraph::op::v8::MaxPool>(ieInpNode, ngraph::Strides(strides), ngraph::Strides(dilations), | ^~ | v7

https://pullrequest.opencv.org/buildbot/builders/4_x_openvino-opencl-skl-lin64/builds/100029

We still use OpenVINO 2021.4

Thanks, I will add a check

OpenVINO backend for INT8 models #23987 ### Pull Request Readiness Checklist TODO: - [x] DetectionOutput layer (#24069) - [x] Less FP32 fallbacks (i.e. Sigmoid, eltwise sum) - [x] Accuracy, performance tests (#24039) - [x] Single layer tests (convolution) - [x] ~~Fixes for OpenVINO 2022.1 (https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100334)~~ Performace results for object detection model `coco_efficientdet_lite0_v1_1.0_quant_2021_09_06.tflite`: | backend | performance (median time) | |---|---| | OpenCV | 77.42ms | | OpenVINO 2023.0 | 10.90ms | CPU: `11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz` Serialized model per-layer stats (note that Convolution should use `*_I8` primitives if they are quantized correctly): https://gist.github.com/dkurt/7772bbf1907035441bb5454f19f0feef --- See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

OpenVINO backend for INT8 models opencv#23987 ### Pull Request Readiness Checklist TODO: - [x] DetectionOutput layer (opencv#24069) - [x] Less FP32 fallbacks (i.e. Sigmoid, eltwise sum) - [x] Accuracy, performance tests (opencv#24039) - [x] Single layer tests (convolution) - [x] ~~Fixes for OpenVINO 2022.1 (https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100334)~~ Performace results for object detection model `coco_efficientdet_lite0_v1_1.0_quant_2021_09_06.tflite`: | backend | performance (median time) | |---|---| | OpenCV | 77.42ms | | OpenVINO 2023.0 | 10.90ms | CPU: `11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz` Serialized model per-layer stats (note that Convolution should use `*_I8` primitives if they are quantized correctly): https://gist.github.com/dkurt/7772bbf1907035441bb5454f19f0feef --- See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

TFLite models on different backends (tests and improvements) opencv#24039 ### Pull Request Readiness Checklist * MaxUnpooling with OpenVINO * Fully connected with transposed inputs/weights with OpenVINO * Enable backends tests for TFLite (related to opencv#23992 (comment)) * Increase existing tests thresholds See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

OpenVINO backend for INT8 models opencv#23987 ### Pull Request Readiness Checklist TODO: - [x] DetectionOutput layer (opencv#24069) - [x] Less FP32 fallbacks (i.e. Sigmoid, eltwise sum) - [x] Accuracy, performance tests (opencv#24039) - [x] Single layer tests (convolution) - [x] ~~Fixes for OpenVINO 2022.1 (https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100334)~~ Performace results for object detection model `coco_efficientdet_lite0_v1_1.0_quant_2021_09_06.tflite`: | backend | performance (median time) | |---|---| | OpenCV | 77.42ms | | OpenVINO 2023.0 | 10.90ms | CPU: `11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz` Serialized model per-layer stats (note that Convolution should use `*_I8` primitives if they are quantized correctly): https://gist.github.com/dkurt/7772bbf1907035441bb5454f19f0feef --- See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

TFLite models on different backends (tests and improvements) opencv#24039 ### Pull Request Readiness Checklist * MaxUnpooling with OpenVINO * Fully connected with transposed inputs/weights with OpenVINO * Enable backends tests for TFLite (related to opencv#23992 (comment)) * Increase existing tests thresholds See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

OpenVINO backend for INT8 models opencv#23987 ### Pull Request Readiness Checklist TODO: - [x] DetectionOutput layer (opencv#24069) - [x] Less FP32 fallbacks (i.e. Sigmoid, eltwise sum) - [x] Accuracy, performance tests (opencv#24039) - [x] Single layer tests (convolution) - [x] ~~Fixes for OpenVINO 2022.1 (https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100334)~~ Performace results for object detection model `coco_efficientdet_lite0_v1_1.0_quant_2021_09_06.tflite`: | backend | performance (median time) | |---|---| | OpenCV | 77.42ms | | OpenVINO 2023.0 | 10.90ms | CPU: `11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz` Serialized model per-layer stats (note that Convolution should use `*_I8` primitives if they are quantized correctly): https://gist.github.com/dkurt/7772bbf1907035441bb5454f19f0feef --- See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dkurt added the category: dnn label Jul 21, 2023

dkurt marked this pull request as ready for review July 24, 2023 11:36

asmorkalov reviewed Jul 25, 2023

View reviewed changes

modules/dnn/src/layers/max_unpooling_layer.cpp Outdated Show resolved Hide resolved

modules/dnn/src/layers/max_unpooling_layer.cpp Outdated Show resolved Hide resolved

asmorkalov self-assigned this Jul 25, 2023

dkurt force-pushed the tflite_test_backends branch from 351cf92 to 5e4f0af Compare July 27, 2023 13:21

asmorkalov requested a review from Abdurrahheem July 27, 2023 13:29

asmorkalov added this to the 4.9.0 milestone Aug 2, 2023

dkurt force-pushed the tflite_test_backends branch from b6152bb to 1899ef0 Compare August 2, 2023 12:46

dkurt mentioned this pull request Aug 2, 2023

OpenVINO backend for INT8 models #23987

Merged

11 tasks

asmorkalov reviewed Aug 3, 2023

View reviewed changes

dkurt added 11 commits August 3, 2023 12:32

Enable backends tests for TFLite

9217f83

Fix OpenVINO resize

8360c96

MaxUnpooling implementation with OpenVINO

8525f93

Update tests

7b1088f

Higher thresholds for FP16 tests

5f1a35f

Skip some MaxPool single layer tests with OpenVINO

49a86d3

Higher thresholds for OCL tests

0493b5f

Enable fully connected layer

24fd2aa

Call getMemoryShapes manually

2b224f6

any axis MatMul with OpenVINO

bd0f1a8

Reshape input of nGraph MatMul

6d66533

Cast MaxUnpool indices if needed

89dee28

dkurt force-pushed the tflite_test_backends branch from 1899ef0 to 89dee28 Compare August 3, 2023 12:31

dkurt commented Aug 3, 2023

View reviewed changes

asmorkalov changed the title ~~TFLite models on different backends (tests and improvements)~~ WIP: TFLite models on different backends (tests and improvements) Aug 3, 2023

dkurt changed the title ~~WIP: TFLite models on different backends (tests and improvements)~~ TFLite models on different backends (tests and improvements) Aug 3, 2023

dkurt mentioned this pull request Aug 3, 2023

Add a test for backends on retrieving intermediate blobs in any order #24102

Closed

4 tasks

asmorkalov approved these changes Aug 4, 2023

View reviewed changes

asmorkalov merged commit 4b8aeb1 into opencv:4.x Aug 4, 2023

asmorkalov mentioned this pull request Aug 7, 2023

(5.x) Merge 4.x #24119

Merged

opencv-alalek reviewed Aug 22, 2023

View reviewed changes

dkurt deleted the tflite_test_backends branch August 23, 2023 07:07

dkurt mentioned this pull request Aug 23, 2023

Enable OpenVINO max pooling with indices since 2022.1 #24189

Merged

6 tasks

Uh oh!

Conversation

dkurt commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

dkurt commented Jul 21, 2023

Uh oh!

opencv-alalek commented Jul 22, 2023

Uh oh!

asmorkalov commented Jul 25, 2023

Uh oh!

asmorkalov commented Jul 25, 2023

Uh oh!

Uh oh!

Uh oh!

dkurt commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Aug 2, 2023

Uh oh!

dkurt commented Aug 2, 2023

Uh oh!

asmorkalov Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Aug 3, 2023

Uh oh!

dkurt commented Aug 3, 2023

Uh oh!

dkurt Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkurt commented Aug 4, 2023

Uh oh!

asmorkalov commented Aug 4, 2023

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

opencv-alalek Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Aug 23, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dkurt commented Jul 21, 2023 •

edited

Loading

dkurt commented Jul 28, 2023 •

edited

Loading

dkurt Aug 3, 2023 •

edited

Loading