core, gapi: supported build with oneTBB 2021 by mshabunin · Pull Request #19384 · opencv/opencv

mshabunin · 2021-01-24T01:22:57Z

resolves #19358

Build Instruction

cmake -DWITH_TBB=ON -DTBB_DIR=<oneTBB>/lib/cmake/tbb ../opencv
# or
TBBROOT=<oneTBB> cmake -DWITH_TBB=ON ../opencv

Sourcing env.sh is not required.

Notes

Disabled oneTBB support in gapi module because of deprecated/removed tbb::task interface being used
Our code relies on TBB_INTERFACE_VER macro defined in tbb_stddef.h to distinguish between versions, but this file has been removed in favor of new version.hpp file. Thus we had to add our own definitions provided via cmake to be able to include correct header: OPENCV_TBB_VERSION_{MAJOR,MINOR}. These two can be replaced with single TBB_USE_VERSION_H definition for better clarity and compactness.
Tested build and smoke test with several official TBB GH releases: 2019-2021 (on Linux)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom
build_image:Custom=ubuntu:20.04
allow_multiple_commits=1

h6197627 · 2021-01-24T12:15:39Z

Also, if possible, I would like to request to fix old problem with TBB (that is also related to oneAPI updates).
When configuring MKL LAPACK to use TBB threading -D MKL_WITH_TBB=ON, TBB search path is wrong for TBB distributions installed from previous Intel Performance Libraries and newer oneAPI distributions
https://github.com/opencv/opencv/blob/master/cmake/OpenCVFindMKL.cmake#L121-L125
In these distributions libraries are placed in ${MKL_ROOT_DIR}/../tbb/lib/${MKL_ARCH}/gcc4.* directories, and without this gcc4.* suffix it fails to find libtbb.so (I am no sure about TBB build from source, maybe there is no additional gcc4.* directories) and fails to configure MKL with TBB threading.

Problem that is directly related to oneAPI TBB is that with this TBB version search path needs to be adjusted
${MKL_ROOT_DIR}/../../tbb/latest/lib/${MKL_ARCH}
(gcc4.* problem is also exists for oneAPI)

mshabunin · 2021-01-24T21:20:17Z

@h6197627 , is tbb actually required to be linked to use MKL TBB backend? As I can see for dynamic libraries it is not needed. Are configurations with static MKL and TBB really exist?

h6197627 · 2021-01-24T21:49:02Z

@mshabunin , I have OpenCV built with TBB and MKL with TBB so I can't say for sure that TBB linkage is not necessary for -D WITH_TBB=OFF -D MKL_WITH_TBB=ON. It might be that this check is redundant.
Interesting that:

ldd libmkl_tbb_thread.so 
        linux-vdso.so.1 (0x00007ffc71fa5000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2147a1f000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2147800000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f214740f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f214972f000)

it is not linked to tbb library (although, when installing TBB threading support component of MKL distribution it installs libtbb.so, so I guess it is not statically linked into libmkl_tbb_thread.so), so it probably uses some sort of runtime dispatch.

Considering, static linkage, I am completely not aware of such usage (although libmkl_tbb_thread.a library exists).

cmake/OpenCVDetectTBB.cmake

alalek · 2021-01-24T23:03:38Z

modules/core/src/parallel.cpp

    #include "tbb/tbb.h"
    #include "tbb/task.h"
+    #if defined(OPENCV_TBB_VERSION_MAJOR) && OPENCV_TBB_VERSION_MAJOR >= 2021
+    #include "tbb/version.h"


This can be detected through try_compile() or checking if(EXISTS ...) / TBB_VER_FILE.

Thank you. Perhaps EXIST check would be the best. I'll update the PR.

Looks like we don't these conditional includes at all.
tbb/version.h or tbb/tbb_stddef.h is included by tbb/tbb.h or by tbb/task.h:

latest public release (2021.1): https://github.com/oneapi-src/oneTBB/blame/onetbb_2021/include/oneapi/tbb.h#L71 (version.h)

2017+: https://github.com/oneapi-src/oneTBB/blame/2017_U1/include/tbb/task.h#L24 (tbb_stddef.h)

2021-beta05 still uses tbb/stddef.h: https://github.com/oneapi-src/oneTBB/commits/88b4c2782a530db1e66947edd2b24699b67850ab/include/tbb/tbb_stddef.h

2021-beta08+ uses version.h: https://github.com/oneapi-src/oneTBB/blame/v2021.1-beta08/include/tbb/tbb.h#L71

P.S. TBB team is not able to work properly with Git history (too many branches are created from clean repo state with lost history)

Removal of these includes works. Tested even on Ubuntu 14.04 with "TBB (ver 4.2 interface 7000)"

mshabunin · 2021-01-26T13:19:34Z

@alalek , I've updated macros mechanism as you suggested.

@h6197627 , I've removed tbb from MKL-TBB dependencies list. Still not sure if it is required, builds with oneAPI without issues (-DWITH_TBB=OFF -DMKL_WITH_TBB=ON). My version of MKL-TBB shows different dependencies list:

$ objdump -p /opt/intel/oneapi/mkl/2021.1.1/lib/intel64/libmkl_tbb_thread.so | grep NEEDED
  NEEDED               libdl.so.2
  NEEDED               libtbb.so.12
  NEEDED               libstdc++.so.6
  NEEDED               libpthread.so.0

Please check on your side if possible.

h6197627 · 2021-01-26T17:59:26Z

@mshabunin ,
interestingly 2021.1.1 has tbb linkage

readelf -d libmkl_tbb_thread.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libtbb.so.12]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]

and 2020.1.217 (which I posted before) does not:

readelf -d libmkl_tbb_thread.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]

alalek · 2021-01-27T17:46:41Z

modules/gapi/src/executor/gtbbexecutor.cpp

 #include "gtbbexecutor.hpp"

-#if defined(HAVE_TBB)
+#if defined(HAVE_TBB) && !defined(OPENCV_TBB_USE_VERSION_H)


Need to include tbb.h / tbb/task.h and then check TBB_INTERFACE_VERSION < XYZ instead.

https://github.com/oneapi-src/oneTBB/blame/v2020.3/include/tbb/tbb_stddef.h#L25

Perhaps XYZ should be < 12000, because -DWITH_TBB=ON -DBUILD_TBB=ON uses 2020.2 with interface 11102 and gapi builds succesfully.

BTW, Please add a comment which API is not available.

alalek · 2021-01-27T18:06:58Z

cmake/OpenCVFindMKL.cmake

  set(mkl_lib_list "mkl_intel_${MKL_ARCH_SUFFIX}")

  if(MKL_WITH_TBB)
-    list(APPEND mkl_lib_list mkl_tbb_thread tbb)


Lets allow to add custom libraries below:

list(APPEND MKL_LIBRARIES ${OPENCV_EXTRA_MKL_LIBRARIES})

alalek · 2021-01-29T06:40:13Z

Validated with:

Intel oneAPI Base Toolkit 2021.1 (on Linux):

. /opt/intel/oneapi/setvars.sh
cmake ../opencv -DWITH_TBB=ON -DCMAKE_DISABLE_FIND_PACKAGE_TBB=1

(CMAKE_DISABLE_FIND_PACKAGE_TBB=1 is necessary to bypass system's CMake scripts (which detects system's tbb) until we get TBBConfig.cmake in oneAPI distribution, probably this will be fixed in the future)

-- Found TBB (env): /opt/intel/oneapi/tbb/2021.1.1/lib/intel64/gcc4.8/libtbb.so
...
-- Found MKL 2021.0.1 at: /opt/intel/oneapi/mkl/latest
-- LAPACK(MKL): LAPACK_LIBRARIES: /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sequential.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sequential.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sequential.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;-lpthread;-lm;-ldl

MKL with TBB case:

cmake ../opencv -DWITH_TBB=ON -DCMAKE_DISABLE_FIND_PACKAGE_TBB=1 -DMKL_WITH_TBB=ON
...
-- Found MKL 2021.0.1 at: /opt/intel/oneapi/mkl/latest
-- LAPACK(MKL): LAPACK_LIBRARIES: /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_tbb_thread.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_tbb_thread.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_tbb_thread.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;-lpthread;-lm;-ldl
-- LAPACK(MKL): Support is enabled.

dmatveev · 2021-02-08T21:04:17Z

Wow, thanks folks!

dizcza · 2023-03-30T07:18:27Z

Please back-port the fix for OpenCV v3.x
I'm cherry-picking your changes...
Thanks

mshabunin · 2023-03-30T10:34:34Z

@dizcza , if you already have cherry-picked commit for 3.4 which you've tested with recent oneTBB, then you can propose it in a PR. It might be faster this way.

alalek reviewed Jan 24, 2021

View reviewed changes

mshabunin force-pushed the support-onetbb branch 2 times, most recently from fde32fb to eaf469b Compare January 26, 2021 13:16

alalek reviewed Jan 27, 2021

View reviewed changes

alalek mentioned this pull request Jan 28, 2021

find mkl and tbb from oneApi automatically. #19415

Closed

6 tasks

mshabunin force-pushed the support-onetbb branch from eaf469b to 181c9f1 Compare January 28, 2021 22:09

mshabunin added 2 commits January 29, 2021 02:38

core, gapi: supported build with oneTBB 2021

694fe3e

MKL-TBB: removed tbb from dependencies list

46b2da4

mshabunin force-pushed the support-onetbb branch from 826c842 to 46b2da4 Compare January 28, 2021 23:39

alalek approved these changes Jan 29, 2021

View reviewed changes

opencv-pushbot merged commit 6be9496 into opencv:master Jan 29, 2021

This was referenced Feb 2, 2021

cmake: workaround oneTBB build issues with MSVC #19447

Merged

cmake: workaround oneAPI problem with -DMKL_WITH_TBB=ON #19451

Merged

mshabunin deleted the support-onetbb branch February 5, 2021 14:29

bayandin mentioned this pull request Mar 15, 2021

tbb 2021.2.0 Homebrew/homebrew-core#72682

Closed

alalek mentioned this pull request Apr 9, 2021

(5.x) Merge 4.x #19885

Merged

carlocab mentioned this pull request May 20, 2021

tbb 2021.2.0 Homebrew/homebrew-core#77662

Closed

alalek mentioned this pull request Jul 11, 2021

Procedure entry point get_initial_auto_partitioner_divisor@internal@tbb@YA_KXZ could not be located in the dynamic link library opencv_core450d.dll #18425

Closed

dizcza mentioned this pull request Jun 15, 2023

[opencv 3.x] back-ported tbb support ubuntu 22.04 #23812

Merged

6 tasks

asmorkalov removed the backport is needed Label for maintainers. Authors of PR can ignore this label Jun 16, 2023

asmorkalov added the port/backport done Label for maintainers. Authors of PR can ignore this label Jun 16, 2023

Uh oh!

Conversation

mshabunin commented Jan 24, 2021 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Instruction

Notes

Pull Request Readiness Checklist

Uh oh!

h6197627 commented Jan 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mshabunin commented Jan 24, 2021

Uh oh!

h6197627 commented Jan 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alalek Jan 24, 2021

Choose a reason for hiding this comment

Uh oh!

mshabunin Jan 24, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jan 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alalek Jan 28, 2021

Choose a reason for hiding this comment

Uh oh!

mshabunin commented Jan 26, 2021

Uh oh!

h6197627 commented Jan 26, 2021

Uh oh!

alalek Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

mshabunin Jan 28, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

mshabunin Jan 28, 2021

Choose a reason for hiding this comment

Uh oh!

alalek commented Jan 29, 2021

Uh oh!

dmatveev commented Feb 8, 2021

Uh oh!

dizcza commented Mar 30, 2023

Uh oh!

mshabunin commented Mar 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mshabunin commented Jan 24, 2021 •

edited by alalek

Loading

h6197627 commented Jan 24, 2021 •

edited

Loading

h6197627 commented Jan 24, 2021 •

edited

Loading

alalek Jan 27, 2021 •

edited

Loading