dnn: TIM-VX NPU backend support by zihaomu · Pull Request #21036 · opencv/opencv

zihaomu · 2021-11-10T14:40:14Z

This PR adds TIM-VX NPU backend to DNN module.

Tutorial on how to use the TIM-VX backend.

https://gist.github.com/fengyuentau/5a7a5ba36328f2b763aea026c43fa45f

Cross compilation for aarch64 and build and run tests with x86_64 simulators using docker: https://gist.github.com/fengyuentau/bf9e54f80dbec4cc0ea13b8f8a985f0b

NPU chip is commonly used DNN acceleration chip. TIM-VX is good NPU backend, it can support 8-bit ops, and its API follows the OpenVX protocol.

Supported Op	status
Conv Int8	Done
Conv Float	Done
Pooling Int8	Done
MatMulInt8	Done
EltwiseInt8	Done
Const	Done
SoftmaxInt8	Partially Done (Only work in ONNX Softmax layer.)
SigmoidInt8	Done
LeakyReluInt8	Done

OpenCV without TIM-VX works with NEON.

	Performance Test on Khadas Vim3
Test Model	Input size	OpenCV with TIM-VX on NPU (ms)	OpenCV without TIM-VX(ms)
Convolution float	1 x 3 x 10 x 10	0.082125	0.011292
Convolution float	1 x 3 x 1280 x 1280	376.457	14.0528
Convolution int8	1 x 3 x 10 x 10	0.053208	0.011625
Convolution int8 (Channel-wise Quantize)	1 x 3 x 1280 x 1280	18.7861	18.7861
Convolution int8 (Tensor-wise Quantize)	1 x 3 x 1280 x 1280	17.01	19.5397
ResNet50 int8 (Channel-wise Quantize)	1 x 3 x 224 x 224	525.298	256.678
ResNet50 int8 (Tensor Wise Quantize)	1 x 3 x 224 x 224	20.01	256.678

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

**WIP**
force_builders_only=linux,Linux OpenCL,Linux AVX2,docs,mac
test_modules=dnn,java,python3

alalek · 2021-11-30T09:49:26Z

cmake/templates/cvconfig.h.in

 #cmakedefine HAVE_VULKAN

+/* TimVx support */
+#cmakedefine HAVE_TIMVX


Don't change cvconfig.h content anymore with module-specific definitions.
Use add_definitions() in DNN module only.

alalek · 2021-11-30T09:50:32Z

modules/dnn/include/opencv2/dnn.hpp

 */
 /** @example samples/dnn/text_detection.cpp
 */
+


remove unnecessary changes from the PR

alalek · 2021-11-30T09:51:30Z

modules/dnn/include/opencv2/dnn/all_layers.hpp

    public:
        int input_zp, output_zp;
-        float output_sc;
+        float output_sc, input_sc;


align order of input/output variables.

alalek · 2021-11-30T09:51:40Z

modules/dnn/include/opencv2/dnn/dict.hpp

    CV_WRAP DictValue(double p)         : type(Param::REAL), pd(new AutoBuffer<double,1>) { (*pd)[0] = p; }     //!< Constructs floating point scalar
    CV_WRAP DictValue(const String &s)  : type(Param::STRING), ps(new AutoBuffer<String,1>) { (*ps)[0] = s; }   //!< Constructs string scalar
    DictValue(const char *s)            : type(Param::STRING), ps(new AutoBuffer<String,1>) { (*ps)[0] = s; }   //!< @overload
+//    DictValue(const uchar *s1)            : type(Param::STRING), ps(new AutoBuffer<String,1>) { (*ps)[0] = s1; }   //!< @overload


alalek · 2021-11-30T09:52:25Z

modules/dnn/include/opencv2/dnn/dnn.hpp

        DNN_TARGET_CUDA_FP16,
-        DNN_TARGET_HDDL
+        DNN_TARGET_NPU,
+        DNN_TARGET_HDDL,


Enum additions should go to the end.

Hi, @alalek. Thank you for code review. In my test, if I move DNN_BACKEND_TIMVX to the end of enum Backend, I will get an error.

If I do the following, the value of DNN_BACKEND_TIMVX in the OpenCV library is 1000001, while the value of DNN_BACKEND_TIMVX when calling the OpenCV library is 6.
This may be caused by #ifdef __OPENCV_BUILD flag. In this case, I can not use DNN_BACKEND_TIMVX to trigger TimVx Backend, I can only use 1000001 to do it.
It happens on Ubuntu of ARM, but not for the Ubuntu of X86.

enum Backend { //! DNN_BACKEND_DEFAULT equals to DNN_BACKEND_INFERENCE_ENGINE if //! OpenCV is built with Intel's Inference Engine library or //! DNN_BACKEND_OPENCV otherwise. DNN_BACKEND_DEFAULT = 0, DNN_BACKEND_HALIDE, DNN_BACKEND_INFERENCE_ENGINE, //!< Intel's Inference Engine computational backend //!< @sa setInferenceEngineBackendType DNN_BACKEND_OPENCV, DNN_BACKEND_VKCOM, DNN_BACKEND_CUDA, #ifdef __OPENCV_BUILD DNN_BACKEND_INFERENCE_ENGINE_NGRAPH = 1000000, // internal - use DNN_BACKEND_INFERENCE_ENGINE + setInferenceEngineBackendType() DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019, // internal - use DNN_BACKEND_INFERENCE_ENGINE + setInferenceEngineBackendType() #endif DNN_BACKEND_TIMVX };

This comment is about TARGET, not BACKEND.

Requirement of addition to the end comes from preserving values of already existed enum items.

In BACKEND put to the end before #ifdef __OPENCV_BUILD

Thank your reply, got it.

alalek · 2021-11-30T09:57:21Z

modules/dnn/src/dnn.cpp

            case DNN_TARGET_FPGA: out << "FPGA"; colorId = 4; break;
            case DNN_TARGET_CUDA: out << "CUDA"; colorId = 5; break;
            case DNN_TARGET_CUDA_FP16: out << "CUDA_FP16"; colorId = 6; break;
+            case DNN_TARGET_NPU: out << "NPU"; colorId = 9; break;


std::vector<string> colors must be updated too.

Thanks, It was fixed in the latest update.

alalek · 2021-11-30T09:58:27Z

modules/dnn/src/int8layers/convolution_layer.cpp


+#ifdef HAVE_TIMVX
+        // For TimVx
+        input_sc = params.get<float>("input_scale");


Field is defined in public API, so it must be initialized regardless of build flags.

alalek · 2021-11-30T09:59:03Z

modules/dnn/src/int8layers/convolution_layer.cpp

        _MM_SET_FLUSH_ZERO_MODE(ftzMode);
        _MM_SET_DENORMALS_ZERO_MODE(dazMode);
 #endif
+


remove unnecessary changes

alalek · 2021-11-30T09:59:40Z

modules/dnn/src/int8layers/eltwise_layer.cpp

        hasVecInput = false;
        op = SUM;
+
+


alalek · 2021-11-30T10:00:58Z

modules/dnn/src/op_timvx.cpp

+    tim::vx::DataType tensorDataType;
+    switch(matDepth)
+    {
+        case 0: // CV_8U


0

we don't need magic numbers here. Use CV_8U directly.

Thanks, it was fixed in the latest update.

fengyuentau

make install will failed when TIM-VX is building from source:

-- Up-to-date: /home/tau/Workspace/fengyuentau/opencv/build/install/lib/libtim-vx.a
-- Up-to-date: /home/tau/Workspace/fengyuentau/opencv/build/install/lib/libtim-vx.a
CMake Error at 3rdparty/libtim-vx/build/src/tim/cmake_install.cmake:78 (file):
  file INSTALL cannot find
  "/home/tau/Workspace/fengyuentau/opencv/include/tim/vx": No such file or
  directory.
Call Stack (most recent call first):
  3rdparty/libtim-vx/build/cmake_install.cmake:42 (include)
  cmake_install.cmake:135 (include)


make: *** [Makefile:86: install] Error 1

We need to find a way to solve this.

3rdparty/libtim-vx/tim-vx.cmake

cmake/OpenCVFindTimVX.cmake

fengyuentau

Current version of cmake code requires TIM-VX to be built as static library, which solves the previous make install error. We will wait for the fix of TIM-VX to solve the linking problem.

3rdparty/libtim-vx/tim-vx.cmake

cmake/OpenCVFindTimVX.cmake

cmake/OpenCVFindTIMVX.cmake

fengyuentau · 2022-01-18T03:25:34Z

@alalek May I know which version of glibc in linux64 in the default ci? We got the following error

/build/precommit_linux64/build/3rdparty/libtim-vx/VIVANTE_SDK/x86_64.prebuilt.ci.sdk/drivers/libOpenVX.so.1: undefined reference to `logf@GLIBC_2.27'
/build/precommit_linux64/build/3rdparty/libtim-vx/VIVANTE_SDK/x86_64.prebuilt.ci.sdk/drivers/libOpenVX.so.1: undefined reference to `powf@GLIBC_2.27'
/build/precommit_linux64/build/3rdparty/libtim-vx/VIVANTE_SDK/x86_64.prebuilt.ci.sdk/drivers/libOpenVX.so.1: undefined reference to `expf@GLIBC_2.27'
collect2: error: ld returned 1 exit status
apps/model-diagnostics/CMakeFiles/opencv_model_diagnostics.dir/build.make:99: recipe for target 'bin/opencv_model_diagnostics' failed
make[2]: Leaving directory '/build/precommit_linux64/build'
make[2]: *** [bin/opencv_model_diagnostics] Error 1
make[1]: *** [apps/model-diagnostics/CMakeFiles/opencv_model_diagnostics.dir/all] Error 2
CMakeFiles/Makefile2:22736: recipe for target 'apps/model-diagnostics/CMakeFiles/opencv_model_diagnostics.dir/all' failed
make[1]: *** Waiting for unfinished jobs....

implying the unmatched versions of glibc between the one used in ci (older) and the one used to compile libOpenVX (newer).

alalek · 2022-01-18T03:49:46Z

Base image for this builder is ubuntu:16.04 (check "init" step logs).
I added other builders based on Ubuntu 18.04

fengyuentau · 2022-01-18T06:32:05Z

Base image for this builder is ubuntu:16.04 (check "init" step logs). I added other builders based on Ubuntu 18.04

@alalek So can I get the same image via docker pull ubuntu:16.04? Verisilicon devs need a same image to compile a SDK that works on linux64 in the ci.

alalek · 2022-01-18T07:57:24Z

the same image

Not the same. Image is based on ubuntu:16.04.
CI images configuration could be found here: https://github.com/opencv-infrastructure/opencv-worker-config/tree/linux/images

Take a look on "Linux AVX2" based on Ubuntu 18.04 - there are set of other issues with pre-compiled binaries.

Pre-compiled binaries are not universal in general. Using of pre-compiled binaries reduces set of supported platforms.
Scope of such platforms should be defined on your side.

alalek · 2022-03-17T16:14:37Z

modules/dnn/src/dnn.cpp


+#ifdef HAVE_TIMVX
+    // update all comsumer
+    void tvUpdateConfictMap(int graphIndex, LayerData& ld, std::vector<std::vector<int> >& graphConflictMap)


To resolve conflicts with merged #21662 please:

move methods declaration into modules/dnn/src/net_impl.hpp file + put timVxInfo declaration near it (after all common fields)

methods implementation should be moved into op_timvx.cpp (see halide code as example)

@alalek Thanks for your guidance. We have completed the code refactoring.

alalek

Overall looks good to me!

Please take a look on "white space" check issues from "Docs" builder.

vpisarev

tested it on my machine, works well!

fengyuentau · 2022-03-24T07:12:50Z

3rdparty/libtim-vx/tim-vx.cmake

+    set(TIMVX_FOUND ON)
+else()
+    set(OCV_TIMVX_FILENAME "${TIMVX_COMMIT_HASH}.zip")
+    set(OCV_TIMVX_URL "https://github.com/fengyuentau/TIM-VX/archive/")


We are going to have a discussion with TIM-VX devs to work out a branch in TIM-VX repo for us to use.

alalek

LGTM 👍

dnn: TIM-VX NPU backend support * Add TimVX NPU backend for DNN module. * use official branch from tim-vx repo; fix detecting viv sdk Co-authored-by: fytao <yuantao.feng@outlook.com>

…or_timvx_backend CI for TIM-VX backend * github actions for TIM-VX backend opencv#21036 * add reference to yuentau/ocv_ubuntu:20.04; remove extra quotes; enable BUILD_TESTS * rename to timvx_backend_tests.yml * add image source prefix * remove if condition for x86_64 simulator

fengyuentau force-pushed the timvx_backend_support branch from c939ea9 to 40a1d74 Compare November 30, 2021 09:12

alalek reviewed Nov 30, 2021

View reviewed changes

fengyuentau reviewed Dec 3, 2021

View reviewed changes

3rdparty/libtim-vx/tim-vx.cmake Outdated Show resolved Hide resolved

cmake/OpenCVFindTimVX.cmake Outdated Show resolved Hide resolved

fengyuentau reviewed Dec 10, 2021

View reviewed changes

3rdparty/libtim-vx/tim-vx.cmake Outdated Show resolved Hide resolved

cmake/OpenCVFindTimVX.cmake Outdated Show resolved Hide resolved

zihaomu force-pushed the timvx_backend_support branch from 559245f to 7e38da5 Compare December 24, 2021 09:44

fengyuentau reviewed Dec 27, 2021

View reviewed changes

cmake/OpenCVFindTIMVX.cmake Outdated Show resolved Hide resolved

zihaomu mentioned this pull request Dec 31, 2021

Add per_tensor_quantize to int8 quantize #21372

Merged

6 tasks

zihaomu marked this pull request as ready for review January 25, 2022 05:11

zihaomu force-pushed the timvx_backend_support branch from 5e0c5dd to 1ae0621 Compare February 14, 2022 10:55

fengyuentau changed the title ~~dnn: TimVx NPU backend support~~ dnn: TIM-VX NPU backend support Feb 27, 2022

zihaomu force-pushed the timvx_backend_support branch from b923992 to 4e2a806 Compare March 4, 2022 10:12

alalek reviewed Mar 17, 2022

View reviewed changes

zihaomu force-pushed the timvx_backend_support branch from 0d5330b to e51bea2 Compare March 18, 2022 06:33

alalek reviewed Mar 18, 2022

View reviewed changes

Add TimVX NPU backend for DNN module.

82eac54

zihaomu force-pushed the timvx_backend_support branch from a00f4a4 to 82eac54 Compare March 18, 2022 11:10

fengyuentau added a commit to fengyuentau/opencv that referenced this pull request Mar 19, 2022

github actions for TIM-VX backend opencv#21036

adc91d3

fengyuentau mentioned this pull request Mar 19, 2022

CI for TIM-VX backend #21752

Merged

6 tasks

vpisarev self-requested a review March 24, 2022 02:09

vpisarev approved these changes Mar 24, 2022

View reviewed changes

vpisarev requested review from alalek and fengyuentau March 24, 2022 02:10

fengyuentau force-pushed the timvx_backend_support branch from 1e0cf9b to 2a7387a Compare March 24, 2022 06:51

fengyuentau reviewed Mar 24, 2022

View reviewed changes

fengyuentau mentioned this pull request Mar 28, 2022

CMake changes for TIM-VX backend in OpenCV VeriSilicon/TIM-VX#336

Merged

zihaomu marked this pull request as ready for review March 31, 2022 02:50

use official branch from tim-vx repo; fix detecting viv sdk

f45400d

fengyuentau force-pushed the timvx_backend_support branch from 23fc2dc to f45400d Compare March 31, 2022 03:11

alalek approved these changes Mar 31, 2022

View reviewed changes

alalek assigned vpisarev Mar 31, 2022

alalek merged commit 7b582b7 into opencv:4.x Mar 31, 2022

opencv-pushbot mentioned this pull request Apr 23, 2022

(5.x) Merge 4.x #21903

Merged

fengyuentau deleted the timvx_backend_support branch May 17, 2022 07:37

Uh oh!

Conversation

zihaomu commented Nov 10, 2021 • edited by fengyuentau Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tutorial on how to use the TIM-VX backend.

Pull Request Readiness Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alalek Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fengyuentau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fengyuentau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fengyuentau commented Jan 18, 2022

Uh oh!

alalek commented Jan 18, 2022

Uh oh!

fengyuentau commented Jan 18, 2022

Uh oh!

alalek commented Jan 18, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

vpisarev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

zihaomu commented Nov 10, 2021 •

edited by fengyuentau

Loading

alalek Dec 1, 2021 •

edited

Loading