Added new data types to cv::Mat & UMat by vpisarev · Pull Request #23865 · opencv/opencv

vpisarev · 2023-06-24T14:37:55Z

This is updated version of #23591.

The related opencv_contrib PR: opencv/opencv_contrib#3523

CV_Bool (boolean), CV_32U, CV_64S, CV_64U and CV_16BF (bfloat16) have been added. Similarly to float16, bfloat16 is always available, not matter if the target architecture supports hardware bf16 ops or not, because conversion to/from FP32 is trivial and super-fast (faster than FP16<=>FP32).
In order to preserve compatibility with other cv::Mat flags, we still use 12 bits for type. The 'depth' bit field now takes 5 bits (allowing for 32 different types), therefore 'channels' field has been shrinked from 9 to 7 bits, allowing 128 channels at maximum (down from 512). Don't worry, in deep learning we use 4D tensors, where channels is a dedicated dimensionality, so this limitation should not affect OpenCV DNN module. For image processing and even multi-spectral image processing 128 channels should probably be enough.
basic operations: copyTo(), setTo(), convertTo(), RNG::fill(), XML, YAML, Json I/O etc. are supported. Arithmetic operations are not extended yet, but at least they don't crash, they throw OpenCV error.

Let's see how many tests fail.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

mshabunin · 2023-06-26T07:30:30Z

modules/core/include/opencv2/core/cvdef.h

 #endif
 };

+class bfloat16_t


It can conflict with C++23 floating-point types: https://en.cppreference.com/w/cpp/types/floating-point , though the same issue can be seen with the float16_t.

thanks for the link, I did not know about those new type definitions. Yes, it would be nice to resolve possible name conflicts.

opencv-alalek

Don't forget to mention related discussions #12584

opencv-alalek · 2023-07-08T23:47:59Z

modules/calib/test/test_multiview_calib.cpp


        std::vector<cv::Mat> img_pts_cams(num_cameras);
-        std::vector<bool> visible(num_cameras, false);
+        std::vector<uchar> visible(num_cameras, (uchar)0);


why?
where is it used with OpenCV API?

the change makes the test pass

opencv-alalek · 2023-07-08T23:52:24Z

modules/core/include/opencv2/core/hal/interface.h

 #define CV_64F  6
 #define CV_16F  7
+#define CV_16BF 8
+#define CV_Bool 9


all uppercase.

P.S. Put #undef in GAPI header.

Finally we should drop defines and start using of C++ enums.

postpone it till the next patch

opencv-alalek · 2023-07-08T23:55:16Z

modules/core/include/opencv2/core/cvdef.h

+    CV_16U - 2 bytes
+    ...
+*/
+#define CV_ELEM_SIZE1(type) ((int)(0x4881228442211ULL >> CV_MAT_DEPTH(type) * 4) & 15)


CV_MAT_DEPTH(type) * 4 => (CV_MAT_DEPTH(type) * 4)

so far there is no warning

opencv-alalek · 2023-07-08T23:58:30Z

modules/core/include/opencv2/core/hal/intrin.hpp

 #endif

+#if (!defined CV_SIMD_64F) || (!CV_SIMD_64F)
+typedef struct v_float64 { int dummy; } v_float64;


It is better to drop this hack,

If we don't have type support, then we don't declare type itself.

disagree; v_float64 and the relevant ops should always be available. CV_SIMD_64F and code fragmentation was a bad idea

opencv-alalek · 2023-07-09T00:00:34Z

modules/core/include/opencv2/core/hal/intrin_avx.hpp

+inline v_int64x4 operator > (const v_int64x4& a, const v_int64x4& b)
+{ return v_int64x4(_mm256_cmpgt_epi64(a.val, b.val)); }
+inline v_int64x4 operator < (const v_int64x4& a, const v_int64x4& b)
+{ return v_int64x4(_mm256_cmpgt_epi64(b.val, a.val)); }


Adding of new intrinsic functions requires modification of C++ reference code / documentation and adding new tests.

ok, sometimes

opencv-alalek · 2023-07-09T00:21:41Z

modules/core/src/persistence.cpp

                        data += sizeof(int);
                        break;
+                    case CV_64U:
+                        ptr = fs::itoa(*(uint64_t*) data, buf, 10, false);


What implementation supports uint64_t?

there is overload

opencv-alalek · 2023-07-09T00:26:12Z

modules/core/src/persistence.cpp

+    const int radix = 10;
+    char* ptr=buffer + 23 /* enough even for 64-bit integers */;
+    int sign = _signed && _val < 0 ? -1 : 1;
+    uint64_t val = _signed ? *(uint64_t*)&_val : abs(_val);


does it really work? condition is wrong

It is better to have different overloads instead.

right, that was a bug; fixed

opencv-alalek · 2023-07-09T00:34:51Z

modules/core/test/test_io.cpp


+    Mat m16bfc1, m16bfc3;
+    m16fc1.convertTo(m16bfc1, CV_16BF);
+    m16fc3.convertTo(m16bfc3, CV_16BF);


test_20279

It is better to write own test function instead of hijacking of existed one.

opencv-alalek · 2023-07-09T00:44:27Z

modules/videoio/src/backend_plugin.cpp

+        if (depth > 7) {
+            // [TODO] temporary hack; remove after rebuilding plugins or add a new
+            // version of plugins. Convert type from the old one to the new one (5 bits)
+            type = CV_MAKETYPE((type & 7), (type >> 3) + 1);


It is time to drop hack on the line 148 of this file.

I guess, it's a subject or a separate PR; feel free to submit it later

opencv-alalek · 2023-07-09T00:54:30Z

modules/calib/test/test_cameracalibration.cpp

        img_pts2.copyTo(image_points_all[1][i]);
    }
    std::vector<Size> image_sizes (2, imageSize);
-    Mat visibility_mat = Mat_<bool>::ones(2, numImgs);


@asmorkalov Need to cleanup this buggy code from 4.x too.

And look on traits why they doesn't disable that.

yes, it's a subject or separate PR

…g randf_64f to exact version used before

…quite strong) assumption that video capture would give us frames with depth == CV_8U (0) or CV_16U (2). If depth is > 7 then it means that the plugin is built with the old OpenCV. It needs to be recompiled, of course and then this hack can be removed.

2. restored ==, !=, > and < univ. intrinsics on ARM32/ARM64.

fengyuentau · 2023-08-04T07:51:40Z

Will this patch be ported back to 4.x?

zihaomu · 2023-08-04T08:10:10Z

Hi @vpisarev, which is our main development branch now?

asmorkalov requested review from mshabunin, opencv-alalek and zihaomu June 25, 2023 09:11

asmorkalov added category: core RFC labels Jun 25, 2023

asmorkalov added this to the 5.0 milestone Jun 25, 2023

mshabunin reviewed Jun 26, 2023

View reviewed changes

opencv-alalek reviewed Jul 9, 2023

View reviewed changes

vpisarev force-pushed the add_new_data_type_v2 branch from 65fe53e to 9805a62 Compare July 11, 2023 06:44

vpisarev added 8 commits July 11, 2023 09:49

started working on adding 32u, 64u, 64s, bool and 16bf types to OpenCV

2f2664a

core & imgproc tests seem to pass

3050416

fixed a few compile errors and test failures on macOS x86

019b974

hopefully fixed some compile problems and test failures

61d53c8

fixed some more warnings and test failures

f945af4

trying to fix small deviations in perf_core & perf_imgproc by reverin…

ff97f97

…g randf_64f to exact version used before

try to repair the case when target arch does not have FP64 SIMD

127fe75

vpisarev force-pushed the add_new_data_type_v2 branch from 9805a62 to 127fe75 Compare July 11, 2023 06:49

1. fixed bug in itoa() found by alalek

7f82ced

2. restored ==, !=, > and < univ. intrinsics on ARM32/ARM64.

vpisarev mentioned this pull request Jul 11, 2023

Added new data types opencv/opencv_contrib#3523

Merged

6 tasks

vpisarev added 6 commits July 11, 2023 14:01

added extra parens

aeecb39

dummy commit to rerun builders

1b972f1

another dummpy commit; rerun macos x64 builder

2c6bc91

attempt to fix the remaining failures and warnings

cd4f62a

another attempt to fix the remaining warning

7d5c0c1

make another likely pointless attempt to fix warning

2d25d77

vpisarev merged commit 518486e into opencv:5.x Aug 4, 2023

opencv-alalek mentioned this pull request Aug 5, 2023

countNonZero returns 286361601 for 112x1 Mat_ #24109

Closed

4 tasks

asmorkalov mentioned this pull request Aug 8, 2023

Fix v_pack_store alignment issue on Windows 32-bit. #24125

Merged

6 tasks

opencv-alalek mentioned this pull request Sep 11, 2023

(5.x) Merge 4.x #24254

Merged

vpisarev deleted the add_new_data_type_v2 branch February 10, 2024 20:46

Uh oh!

Conversation

vpisarev commented Jun 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opencv-alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fengyuentau commented Aug 4, 2023

Uh oh!

zihaomu commented Aug 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vpisarev commented Jun 24, 2023 •

edited

Loading

zihaomu commented Aug 4, 2023 •

edited

Loading