Added clapack by vpisarev · Pull Request #18571 · opencv/opencv

vpisarev · 2020-10-12T00:15:36Z

build_image:Docs=docs-js
force_builders=Linux AVX2,Win32,Linux32,ARMv7,ARMv8
buildworker:Mac=macosx-1
buildworker:iOS=macosx-1
buildworker:Linux x64 Debug=linux-1

The patch adds CLapack back to OpenCV (let's call it "built-in Lapack"). Although OpenCV can make use of external optimized versions of Lapack (e.g. MKL on Intel platforms, Accelerate framework on Apple devices), in which case the built-in Lapack is not used, having builtin Lapack is useful because we can enable by default all the algorithms that depend on it, e.g. the new RANSAC (and thus remove completely the old RANSAC)

Pull Request Readiness Checklist

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

References

Evolution proposal to bring Lapack back to OpenCV: https://github.com/opencv/opencv/wiki/OE-12.-Lapack
Feature request to add eigenvalue solver for non-symmetric matrices: Feature request: eigenNonSymmetric() with complex eigenvalues #16298
Bug report about slow down in OpenCV's SVD: OpenCV 3.1's SVD is 3600 times slower than OpenCV 2.2's SVD #10084

alalek · 2020-10-13T11:18:05Z

The following accurate information is necessary to maintain any 3rdparty library:

exact location and version of upsteam library code
what patch is applied over upstream source code? (necessary for regular upstream updates, it should be minimal)
what is real coverage of these 120k LOC? Does OpenCV really utilizes all this code? only minimal subset of upstream library should be included.

vpisarev · 2020-10-13T13:11:52Z

@alalek, I will provide some information in README.
Here I put it as well:

the code is taken from OpenCV 2.2. It's CLapack 3.2.1 (the main site is not responding currently, here is one of the multiple mirrors: https://github.com/hunter-packages/clapack, the last semi-automatically generated C version of Lapack (https://github.com/Reference-LAPACK/lapack). Lapack is now at 3.9.0. But since Lapack 3.3.0 they started to use some newer standard of Fortran, which is not supported by f2c. I tried to run 3.9.0 through f2c nevertheless, most of the files have been converted successfully, especially from the subset that we use in OpenCV. I think, f2c can be rather easily patched to support the newer versions of Lapack. I plan to upgrade CLapack 3.2.1 to "CLapack" 3.9.0 (or the later actual version) next year. Since the conversion is very fast, we can even store .f sources in OpenCV repository and convert them as a part of compile step.
yes, the changes that we need are rather local, like custom f2c.h header, heavily reduced cblas.h and somewhat changed lapacke.h, as well as some patches in blas functions, like gemm and xerbla. I'll try to make them more modular.
unfortunately, this is close to the minimum possible subset of Lapack (i.e. the coverage is pretty good). For the reference, complete CLapack + CBlas from netlib consists of ~1900 source files; this patch includes ~350 files, which is just 18%. The external functions that we use or can use are GEMM, LU, Cholesky, QR, SVD and Eigen decompositions; in the latter case there are 2 separate big branches for symmetrical and general matrices. I started with the files that are really needed and gradually added functions from CLapack that the linker complained about. This procedure can and should be automated, of course.

…enCV

* trying to fix remaining CI problems

…ositions. It adds very little extra overhead. * added stub version of sdesdd. * uncommented calls to all the single-precision Lapack functions from opencv/core/src/hal_internal.cpp.

vpisarev · 2020-10-19T15:46:13Z

reduced subset of added CLapack even further. The number of files decreased from 300+ down to 70. The only sacrifice is sgesdd_ (single-precision SVD). The patch size is now ≈700Kb and the generated libclapack.a is also ≈700Kb (comparable to libjpeg-turbo or libwebp).
updated CLapack from 3.2.1 to the latest one, generated from Lapack 3.9.0 (released on 2019 Nov).
added little Python script make_clapack.py to easily build a subset of CLapack from Lapack when necessary.
updated f2c compiler to handle the new Lapack sources correctly and generate more convenient for integration code (hosted in a separate repository https://github.com/vpisarev/f2c/tree/for_lapack)

…(such as sqrt(), r_cnjg() etc.) * at once, trailing whitespaces are removed from the generated sources, just in case * since there is no declarations of intrinsic functions anymore, we could turn some of them into inline functions

…standard LAPACK detection procedure * removed some more trailing whitespaces

vpisarev · 2020-10-20T11:39:59Z

@alalek, please review. I've finished the work on it. "Linux x64 Debug" build slave fails sometimes due to unknown reason on various pull requests. It needs to be rerun, perhaps

* added checks to lapack calls to gracefully return "not implemented" instead of returning invalid results with "ok" status

alalek

Looks good to me 👍

opencv-alalek · 2024-06-13T12:00:54Z

modules/core/src/hal_internal.cpp

 {
    int lda = (int)(a_step / sizeof(fptype)), sign = 0;
-    int* piv = new int[m];
+    std::vector<int> piv(m+1);


Why do we need +1 here?

If it is a bug, then this bugfix was NOT backported to 4.x.

/cc @vpisarev @asmorkalov

@opencv-alalek, @asmorkalov
maybe it's not relevant anymore. As I remember, it's just common pattern to handle m=0 case properly.

opencv-alalek · 2024-11-27T05:57:56Z

3rdparty/clapack/runtime/cblas_wrap.c

+      fprintf(stderr, "Parameter %d to routine %s was incorrect\n", info, rout);
+   vfprintf(stderr, form, argptr);
+   va_end(argptr);
+   if (info && !info)


Looks like a bug: #26528

exit(-1)

Should not have that.

bring a small subset of Lapack, automatically converted to C, into Op…

b03aa65

…enCV

vpisarev force-pushed the add_lapack branch from 0b7d268 to b03aa65 Compare October 19, 2020 07:25

vpisarev added 5 commits October 19, 2020 10:26

added missing lsame_ prototype

96b73aa

* small fix in make_clapack script

d187022

* trying to fix remaining CI problems

fixed character arrays' initializers

f86950b

get rid of F2C_STR_MAX

bbdba6a

* added back single-precision versions for QR, LU and Cholesky decomp…

2109082

…ositions. It adds very little extra overhead. * added stub version of sdesdd. * uncommented calls to all the single-precision Lapack functions from opencv/core/src/hal_internal.cpp.

vpisarev added 5 commits October 19, 2020 20:34

fixed warning from Visual Studio + cleaned f2c runtime a bit

048cf2b

trying to eliminate the crash on ARM

d9748d3

fixed API and semantics of s_copy

841e186

* CLapack has been tested successfully. It's now time to restore the …

bcb0193

…standard LAPACK detection procedure * removed some more trailing whitespaces

vpisarev and others added 3 commits November 4, 2020 01:03

* retained only the essential stuff in CLapack

fcd28cb

* added checks to lapack calls to gracefully return "not implemented" instead of returning invalid results with "ok" status

disabled warning when building lapack

082c847

cmake: update LAPACK detection

3368a17

alalek approved these changes Nov 5, 2020

View reviewed changes

alalek merged commit 2ee9d21 into opencv:next Nov 5, 2020

vpisarev mentioned this pull request Nov 9, 2020

OE-12 "Bring Lapack back" discussion #11007

Closed

vpisarev deleted the add_lapack branch November 9, 2020 20:51

vpisarev mentioned this pull request May 25, 2021

Fixed detection of OpenBLAS and LAPACK on Ubuntu 18.04 with arch x86-64 #19343

Open

6 tasks

opencv-alalek reviewed Jun 13, 2024

View reviewed changes

opencv-alalek mentioned this pull request Nov 27, 2024

build: disable clang warning in clapack 3rdparty #26528

Merged

opencv-alalek reviewed Nov 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added clapack#18571

Added clapack#18571
alalek merged 14 commits intoopencv:nextfrom
vpisarev:add_lapack

vpisarev commented Oct 12, 2020 •

edited by alalek

Loading

Uh oh!

alalek commented Oct 13, 2020

Uh oh!

vpisarev commented Oct 13, 2020 •

edited

Loading

Uh oh!

vpisarev commented Oct 19, 2020 •

edited

Loading

Uh oh!

vpisarev commented Oct 20, 2020

Uh oh!

alalek left a comment

Uh oh!

opencv-alalek Jun 13, 2024

Uh oh!

vpisarev Jun 25, 2024 •

edited

Loading

Uh oh!

opencv-alalek Nov 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vpisarev commented Oct 12, 2020 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

References

Uh oh!

alalek commented Oct 13, 2020

Uh oh!

vpisarev commented Oct 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vpisarev commented Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vpisarev commented Oct 20, 2020

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

opencv-alalek Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

vpisarev Jun 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opencv-alalek Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vpisarev commented Oct 12, 2020 •

edited by alalek

Loading

vpisarev commented Oct 13, 2020 •

edited

Loading

vpisarev commented Oct 19, 2020 •

edited

Loading

vpisarev Jun 25, 2024 •

edited

Loading