FastCV-based HAL for OpenCV acceleration by sssanjee-quic · Pull Request #26316 · opencv/opencv

sssanjee-quic · 2024-10-16T12:23:21Z

Added Fastcv HAL changes in the 3rdparty folder.
Code Changes includes HAL code , Fastcv libs and Headers

Change-Id: I2f0ddb1f57515c82ae86ba8c2a82965b1a9626ec

Requires binaries from opencv/opencv_3rdparty#86.
Related patch to opencv_contrib: opencv/opencv_contrib#3811

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Added Fastcv HAL changes in the 3rdparty folder. Includes HAL code , Fastcv libs and Headers Change-Id: I2f0ddb1f57515c82ae86ba8c2a82965b1a9626ec

3rdparty/fastcv/CMakeLists.txt

CMakeLists.txt

asmorkalov · 2024-10-22T13:38:51Z

@sssanjee-quic Please create dedicated PR with binaries in opencv_3rdparty repository as we discussed. Static libraries are preferable instead of the dynamic ones.

3rdparty/fastcv/hal/CMakeLists.txt

3rdparty/fastcv/hal/fastcv_hal_utils.cpp

3rdparty/fastcv/hal/fastcv_hal_core.cpp

savuor · 2024-10-24T00:40:19Z

3rdparty/fastcv/hal/fastcv_hal_imgproc.cpp

+#include <opencv2/core/utils/logger.hpp>
+#include <opencv2/imgproc.hpp>
+
+extern bool isInitialized;


This can be moved to fastcv_hal_utils.hpp

will update in next PR

3rdparty/fastcv/hal/fastcv_hal_core.cpp

CMakeLists.txt

Reorganized the files in the folder, Added rules to download the fastcv bins from the 3rdparty repo Change-Id: Ia66f138894d3e8a06d0f32a6e7ce3359619189d7

3rdparty/fastcv/include/fastcv_hal_utils.hpp

3rdparty/fastcv/src/fastcv_hal_core.cpp

3rdparty/fastcv/src/fastcv_hal_imgproc.cpp

3rdparty/fastcv/src/fastcv_hal_core.cpp

asmorkalov · 2024-10-29T11:43:45Z

There is some bug in CMake. FastCV is downloaded and reported in CMake status, but HAL is not built and linked:
CMake command for Android:

cmake -DCMAKE_TOOLCHAIN_FILE=/home/alexander/Android/Sdk/ndk/28.0.12433566/build/cmake/android.toolchain.cmake -DWITH_KLEIDICV=OFF -DANDROID_NATIVE_API_LEVEL=24 -DANDROID_NDK=/home/alexander/Android/Sdk/ndk/28.0.12433566/ -DANDROID_SDK=/home/alexander/Android/Sdk/ -DANDROID_ABI="arm64-v8a" ../opencv

CMake status output:

  Other third-party libraries:
    Custom HAL:                  YES (fastcv (ver 0.0.1) carotene (ver 0.0.1))
    Protobuf:                    build (3.19.1)
    Flatbuffers:                 builtin/3rdparty (23.5.9)

Actual binaries do not have FastCV dependency:

readelf -d ./bin/opencv_test_core 

Dynamic section at offset 0x929cb8 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000007 (RELA)               0x2110
 0x0000000000000008 (RELASZ)             576000 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          23983
 0x0000000000000017 (JMPREL)             0x8eb10
 0x0000000000000002 (PLTRELSZ)           4872 (bytes)
 0x0000000000000003 (PLTGOT)             0x92e658
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x348
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x18f4
 0x000000000000000a (STRSZ)              2073 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x18d8
 0x0000000000000019 (INIT_ARRAY)         0x92dad8
 0x000000000000001b (INIT_ARRAYSZ)       480 (bytes)
 0x000000006ffffff0 (VERSYM)             0x16c8
 0x000000006ffffffe (VERNEED)            0x1868
 0x000000006fffffff (VERNEEDNUM)         3
 0x0000000000000000 (NULL)               0x0

for java part:

readelf -d ./jni/arm64-v8a/libopencv_java4.so 

Dynamic section at offset 0x1552788 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libjnigraphics.so]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so]
 0x0000000000000001 (NEEDED)             Shared library: [libandroid.so]
 0x0000000000000001 (NEEDED)             Shared library: [libmediandk.so]
 0x0000000000000001 (NEEDED)             Shared library: [libcamera2ndk.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000e (SONAME)             Library soname: [libopencv_java4.so]
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x0000000000000007 (RELA)               0xc6740
 0x0000000000000008 (RELASZ)             1240752 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          38680
 0x0000000000000017 (JMPREL)             0x1f55f0
 0x0000000000000002 (PLTRELSZ)           63336 (bytes)
 0x0000000000000003 (PLTGOT)             0x15581b0
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x2f8
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x4e9ec
 0x000000000000000a (STRSZ)              490833 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x3efe0
 0x0000000000000019 (INIT_ARRAY)         0x1556608
 0x000000000000001b (INIT_ARRAYSZ)       384 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x15565f8
 0x000000000000001c (FINI_ARRAYSZ)       16 (bytes)
 0x000000006ffffff0 (VERSYM)             0x3a228
 0x000000006ffffffe (VERNEED)            0x3ef6c
 0x000000006fffffff (VERNEEDNUM)         3
 0x0000000000000000 (NULL)               0x0

asmorkalov · 2024-11-18T09:06:49Z

There is conflict with last KleidiCV-related patch. Could you rebase on top of 4.x and fix the conflict?

asmorkalov · 2024-11-18T09:07:10Z

Build warnings with fresh NDK:

[493/1895] Building CXX object 3rdparty/fastcv/CMakeFiles/fastcv_hal.dir/src/fastcv_hal_core.cpp.o
/mnt/Projects/Projects/opencv/3rdparty/fastcv/src/fastcv_hal_core.cpp:112:40: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'int' [-Wsign-compare]
  112 |     if (src1_step < width && src2_step < width)
      |                              ~~~~~~~~~ ^ ~~~~~
/mnt/Projects/Projects/opencv/3rdparty/fastcv/src/fastcv_hal_core.cpp:112:19: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'int' [-Wsign-compare]
  112 |     if (src1_step < width && src2_step < width)
      |         ~~~~~~~~~ ^ ~~~~~
/mnt/Projects/Projects/opencv/3rdparty/fastcv/src/fastcv_hal_core.cpp:139:40: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'int' [-Wsign-compare]
  139 |     if (src1_step < width && src2_step < width)
      |                              ~~~~~~~~~ ^ ~~~~~
/mnt/Projects/Projects/opencv/3rdparty/fastcv/src/fastcv_hal_core.cpp:139:19: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'int' [-Wsign-compare]
  139 |     if (src1_step < width && src2_step < width)
      |         ~~~~~~~~~ ^ ~~~~~
/mnt/Projects/Projects/opencv/3rdparty/fastcv/src/fastcv_hal_core.cpp:199:23: warning: unused parameter 'mask_step' [-Wunused-parameter]
  199 |     size_t            mask_step)

The last one looks like some bug. OpenCV allows noncontinuous matrices for mask too.

quic-xuezha · 2024-11-18T09:58:42Z

There is conflict with last KleidiCV-related patch. Could you rebase on top of 4.x and fix the conflict?

done

3rdparty/fastcv/src/fastcv_hal_core.cpp

3rdparty/fastcv/CMakeLists.txt

3rdparty/fastcv/fastcv.cmake

quic-xuezha · 2024-11-20T09:01:06Z

Hi @asmorkalov the warnings we will fix, but the error about the doc, how should we fix it? #7601

asmorkalov · 2024-11-20T10:29:18Z

Please ignore patch size check. The result looks like false-alarm.

asmorkalov · 2024-12-02T07:51:51Z

@sssanjee-quic Thanks a lot for the great job! The PR has been squashed and merged manually through #26556

Depends on opencv/opencv#26316 ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [ ] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

OpenCV Acceleration with FastCV HAL changes

ca32a3a

Added Fastcv HAL changes in the 3rdparty folder. Includes HAL code , Fastcv libs and Headers Change-Id: I2f0ddb1f57515c82ae86ba8c2a82965b1a9626ec

asmorkalov changed the title ~~OpenCV Acceleration with FastCV HAL changes~~ FastCV-based HAL for OpenCV acceleration Oct 16, 2024

asmorkalov reviewed Oct 16, 2024

View reviewed changes

3rdparty/fastcv/CMakeLists.txt Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

asmorkalov added optimization platform: android RFC platform: linux platform: arm ARM boards related issues: RPi, NVIDIA TK/TX, etc category: 3rdparty labels Oct 16, 2024

asmorkalov self-assigned this Oct 16, 2024

asmorkalov added this to the 4.11.0 milestone Oct 16, 2024

savuor reviewed Oct 23, 2024

View reviewed changes

3rdparty/fastcv/hal/CMakeLists.txt Outdated Show resolved Hide resolved

savuor reviewed Oct 23, 2024

View reviewed changes

3rdparty/fastcv/hal/fastcv_hal_utils.cpp Outdated Show resolved Hide resolved

savuor reviewed Oct 23, 2024

View reviewed changes

3rdparty/fastcv/hal/fastcv_hal_utils.cpp Outdated Show resolved Hide resolved

savuor reviewed Oct 23, 2024

View reviewed changes

3rdparty/fastcv/hal/fastcv_hal_utils.cpp Outdated Show resolved Hide resolved

savuor reviewed Oct 24, 2024

View reviewed changes

3rdparty/fastcv/hal/fastcv_hal_core.cpp Show resolved Hide resolved

savuor reviewed Oct 24, 2024

View reviewed changes

3rdparty/fastcv/hal/fastcv_hal_core.cpp Outdated Show resolved Hide resolved

savuor reviewed Oct 24, 2024

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

Fastcv HAL changes for Opencv Acceleration

ccbe833

Reorganized the files in the folder, Added rules to download the fastcv bins from the 3rdparty repo Change-Id: Ia66f138894d3e8a06d0f32a6e7ce3359619189d7

asmorkalov reviewed Oct 29, 2024

View reviewed changes

asmorkalov removed the RFC label Nov 13, 2024

qc-apreetam and others added 4 commits November 14, 2024 16:28

Added Singleton Class

0b0d731

Code review fixes: Reworked CMake scripts.

5e3a9c1

return NOT_IMPLEMENTED when unused parameters are not default value

4be6eeb

Add library for Linux

f89e149

Merge remote-tracking branch 'github_opencv/4.x' into FastcvHAL_1stPost

e5497bf

quic-xuezha reviewed Nov 18, 2024

View reviewed changes

3rdparty/fastcv/src/fastcv_hal_core.cpp Show resolved Hide resolved

asmorkalov mentioned this pull request Nov 19, 2024

FastCV Extension code for Opencv opencv/opencv_contrib#3824

Merged

6 tasks

asmorkalov reviewed Nov 20, 2024

View reviewed changes

3rdparty/fastcv/CMakeLists.txt Outdated Show resolved Hide resolved

3rdparty/fastcv/CMakeLists.txt Outdated Show resolved Hide resolved

3rdparty/fastcv/fastcv.cmake Show resolved Hide resolved

adsha-quic and others added 3 commits November 21, 2024 14:29

fix warnings in meanStdDev and multiply,sub arithm mixed hals

b39dc42

Update cmake files to get PR ready to merge

3d0f56b

Change FastCV hal order

ea14b9e

asmorkalov approved these changes Nov 28, 2024

View reviewed changes

asmorkalov closed this Dec 2, 2024

Uh oh!

Conversation

sssanjee-quic commented Oct 16, 2024 • edited by asmorkalov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Oct 22, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

savuor Oct 24, 2024

Choose a reason for hiding this comment

Uh oh!

ghost Oct 25, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Oct 29, 2024

Uh oh!

asmorkalov commented Nov 18, 2024

Uh oh!

asmorkalov commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

quic-xuezha commented Nov 18, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

quic-xuezha commented Nov 20, 2024

Uh oh!

asmorkalov commented Nov 20, 2024

Uh oh!

asmorkalov commented Dec 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sssanjee-quic commented Oct 16, 2024 •

edited by asmorkalov

Loading

asmorkalov commented Nov 18, 2024 •

edited

Loading