Added int support for OpenVINO dnn backend by alexlyulkov · Pull Request #25458 · opencv/opencv

alexlyulkov · 2024-04-19T10:18:59Z

Modified dnn OpenVINO integration to support type inference and int operations.

Added OpenVINO support to Cast, CumSum, Expand, Gather, GatherElements, Scatter, ScatterND, Tile layers.
I tried to add Reduce layer, but looks like OpenVINO uses float values inside Reduce operation so it can't pass our int tests.

OpenVINO uses int32 precision for int64 operations, so I've modified input values for int64 tests when backend is OpenVINO.

OpenVINO has a strange behavior with custom layers and int64 values. After model compilation OpenVINO may change types, so the model can have different output type. That's why these tests were disabled:

Test_ArgMax_Int.random/0, where GetParam() = (4, NGRAPH/CPU)
Test_ArgMax_Int.random/6, where GetParam() = (11, NGRAPH/CPU)
Test_Reduce_Int.random/6, where GetParam() = (11, NGRAPH/CPU)
Test_Reduce_Int.two_axes/6, where GetParam() = (11, NGRAPH/CPU)

Also these tests were temporary disabled, they didn't work on both 4.x and 5.x branches:

Test_Caffe_layers.layer_prelu_fc/0, where GetParam() = NGRAPH/CPU
Test_ONNX_layers.LSTM_Activations/0, where GetParam() = NGRAPH/CPU
Test_ONNX_layers.Quantized_Convolution/0, where GetParam() = NGRAPH/CPU
Test_ONNX_layers.Quantized_Eltwise_Scalar/0, where GetParam() = NGRAPH/CPU
Test_TFLite.EfficientDet_int8/0, where GetParam() = NGRAPH/CPU

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

modules/dnn/src/layers/blank_layer.cpp

modules/dnn/src/ie_ngraph.cpp

modules/dnn/src/net_impl.cpp

modules/dnn/src/layers/expand_layer.cpp

modules/dnn/src/ie_ngraph.cpp

asmorkalov · 2024-04-26T08:25:51Z

[ RUN      ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_negative_indices_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_negative_indices, NGRAPH/CPU)
Exception during net.forward() call!
unknown file: Failure
C++ exception with description "Check 'false' failed at src/inference/src/infer_request.cpp:215:
ScatterElementsUpdate node with name 'y' have indices value that points to non-existing output tensor element
" thrown in the test body.
[  FAILED  ] Test_ONNX_conformance.Layer_Test/test_scatter_elements_with_negative_indices_NGRAPH_CPU, where GetParam() = (test_scatter_elements_with_negative_indices, NGRAPH/CPU) (10 ms)

asmorkalov · 2024-04-26T08:26:22Z

Other failures are related to CI. I'll fix it and return back today.

modules/dnn/test/test_onnx_importer.cpp

dkurt · 2024-04-26T13:50:09Z

modules/dnn/test/test_tflite_importer.cpp


 TEST_P(Test_TFLite, EfficientDet_int8) {
+    if (backend == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH)
+        applyTestTag(CV_TEST_TAG_DNN_SKIP_IE_NGRAPH); // TODO: fix this test for OpenVINO


Did test work before the patch?

This test doesn't work in both 4.x and 5.x branches. Probably because I use the newest OpenVINO version

Test definitely worked on 2023 release of OpenVINO. I will check 2024 version. I think ok to skip on this PR.

asmorkalov · 2024-05-13T12:56:19Z

Backport: #25524

asmorkalov · 2024-05-15T08:04:24Z

@dkurt could you take a look again?

Added int support for OpenVINO dnn backend(draft)

5f08d44

alexlyulkov requested review from asmorkalov and dkurt April 19, 2024 10:18

asmorkalov added the category: dnn label Apr 19, 2024

asmorkalov added this to the 5.0 milestone Apr 19, 2024

Added more OpenVINO layers, fixed several problems

0fc4564

alexlyulkov commented Apr 25, 2024

View reviewed changes

modules/dnn/src/layers/blank_layer.cpp Show resolved Hide resolved

Minor fix

1ccab2f

alexlyulkov marked this pull request as ready for review April 25, 2024 16:52

dkurt reviewed Apr 25, 2024

View reviewed changes

modules/dnn/src/ie_ngraph.cpp Show resolved Hide resolved