Skip to content

Use cvtColor() for Bayer image color demosaicing with V4L2_PIX_FMT_SGRBG8#25287

Merged
asmorkalov merged 1 commit intoopencv:4.xfrom
catree:feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor
Apr 4, 2024
Merged

Use cvtColor() for Bayer image color demosaicing with V4L2_PIX_FMT_SGRBG8#25287
asmorkalov merged 1 commit intoopencv:4.xfrom
catree:feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor

Conversation

@catree
Copy link
Copy Markdown
Contributor

@catree catree commented Mar 28, 2024

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

See: #25249

@carabsc Does this change works for you?
If so, I believe it is better to use cv::cvtColor() function.

Warning, this is GRBG --> BGR.

BTW, can we do the same changes with

@catree
Copy link
Copy Markdown
Contributor Author

catree commented Mar 28, 2024

Looking at the code more closely:

  • case V4L2_PIX_FMT_SBGGR8:
    bayer2rgb24(imageSize.width, imageSize.height,
    start, (unsigned char*)frame.imageData);
    return;
  • /* B */
    if ( (i > WIDTH) && ((i % WIDTH) > 0) ) {
    *scanpt++ = (*(rawpt-WIDTH-1)+*(rawpt-WIDTH+1)+
    *(rawpt+WIDTH-1)+*(rawpt+WIDTH+1))/4; /* R */
    *scanpt++ = (*(rawpt-1)+*(rawpt+1)+
    *(rawpt+WIDTH)+*(rawpt-WIDTH))/4; /* G */
    *scanpt++ = *rawpt; /* B */

These lines of code do BGGR --> RGB. You can see it in the case of even i and j indices: B output value is directly the value of *scanpt++ = *rawpt;:

image


But with V4L2_PIX_FMT_SGBRG8:

  • case V4L2_PIX_FMT_SGBRG8:
    sgbrg2rgb24(imageSize.width, imageSize.height,
    start, (unsigned char*)frame.imageData);
    return;
  • if ( (i/WIDTH) % 2 == 0 ) //even row
    {
    if ( (i % 2) == 0 ) //even pixel
    {
    if ( (i > WIDTH) && ((i % WIDTH) > 0) )
    {
    *scanpt++ = (*(rawpt-1)+*(rawpt+1))/2; /* R */
    *scanpt++ = *(rawpt); /* G */
    *scanpt++ = (*(rawpt-WIDTH) + *(rawpt+WIDTH))/2; /* B */

Here we have instead GBRG --> BGR:

image

With even i and j, the output B component is the average from left and right pixels.


To sum-up:

  • V4L2_PIX_FMT_SBGGR8 and V4L2_PIX_FMT_SN9C10X do BGGR --> RGB
  • while V4L2_PIX_FMT_SGBRG8 do GBRG --> BGR
  • and V4L2_PIX_FMT_SGRBG8 do GRBG --> BGR

@catree catree force-pushed the feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor branch from 8aa6e12 to ec16cb7 Compare March 28, 2024 23:12
@asmorkalov asmorkalov added category: videoio cleanup Code cleanup (e.g, drop legacy C-API, legacy unmaintained code) labels Mar 29, 2024
@asmorkalov asmorkalov added this to the 4.10.0 milestone Mar 29, 2024
@catree catree force-pushed the feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor branch from ec16cb7 to eaa01f2 Compare March 30, 2024 12:58
@asmorkalov
Copy link
Copy Markdown
Contributor

@catree go ahead with other cases.

@catree catree force-pushed the feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor branch from eaa01f2 to 4061b1d Compare April 2, 2024 11:56
@catree
Copy link
Copy Markdown
Contributor Author

catree commented Apr 2, 2024

@asmorkalov

Now we have the following cases which use cvtColor() function:

  • V4L2_PIX_FMT_SN9C10X --> BGR (breaking change, before it was to RGB)
  • V4L2_PIX_FMT_SRGGB8 (added) --> BGR
  • V4L2_PIX_FMT_SBGGR8, V4L2_PIX_FMT_SGBRG8 and V4L2_PIX_FMT_SGRBG8 --> BGR

@catree catree force-pushed the feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor branch from 4061b1d to 4525060 Compare April 2, 2024 12:14
@mshabunin
Copy link
Copy Markdown
Contributor

I'm wondering whether these modes can be added/enabled to our virtual V4L device test:

vector<Format_Channels_Depth> all_params = {
{ V4L2_PIX_FMT_YVU420, 1, CV_8U, 1.f, 1.5f },
{ V4L2_PIX_FMT_YUV420, 1, CV_8U, 1.f, 1.5f },
{ V4L2_PIX_FMT_NV12, 1, CV_8U, 1.f, 1.5f },
{ V4L2_PIX_FMT_NV21, 1, CV_8U, 1.f, 1.5f },
{ V4L2_PIX_FMT_YUV411P, 3, CV_8U, 1.f, 1.f },
// { V4L2_PIX_FMT_MJPEG, 1, CV_8U, 1.f, 1.f },
// { V4L2_PIX_FMT_JPEG, 1, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_YUYV, 2, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_UYVY, 2, CV_8U, 1.f, 1.f },
// { V4L2_PIX_FMT_SBGGR8, 1, CV_8U, 1.f, 1.f },
// { V4L2_PIX_FMT_SN9C10X, 3, CV_8U, 1.f, 1.f },
// { V4L2_PIX_FMT_SGBRG8, 1, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_RGB24, 3, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_Y16, 1, CV_16U, 1.f, 1.f },
{ V4L2_PIX_FMT_Y16_BE, 1, CV_16U, 1.f, 1.f },
{ V4L2_PIX_FMT_Y10, 1, CV_16U, 1.f, 1.f },
{ V4L2_PIX_FMT_GREY, 1, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_BGR24, 3, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_XBGR32, 3, CV_8U, 1.f, 1.f },
{ V4L2_PIX_FMT_ABGR32, 3, CV_8U, 1.f, 1.f },
};

@catree catree force-pushed the feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor branch from 4525060 to c10d0c2 Compare April 3, 2024 07:49
@catree
Copy link
Copy Markdown
Contributor Author

catree commented Apr 3, 2024

I have updated the test file.

@mshabunin
Copy link
Copy Markdown
Contributor

Could you please also add something like this to the test:

diff --git a/modules/videoio/test/test_v4l2.cpp b/modules/videoio/test/test_v4l2.cpp
index 12d2a7210e..5bb1a1f5a3 100644
--- a/modules/videoio/test/test_v4l2.cpp
+++ b/modules/videoio/test/test_v4l2.cpp
@@ -70,6 +70,7 @@ TEST_P(videoio_v4l2, formats)
     const string device = devs[0];
     const Size sz(640, 480);
     const Format_Channels_Depth params = GetParam();
+    const Size esz(sz.width * params.mul_width, sz.height * params.mul_height);
 
     {
         // Case with RAW output
@@ -83,7 +84,17 @@ TEST_P(videoio_v4l2, formats)
             Mat img;
             EXPECT_TRUE(cap.grab());
             EXPECT_TRUE(cap.retrieve(img));
-            EXPECT_EQ(Size(sz.width * params.mul_width, sz.height * params.mul_height), img.size());
+            if (params.pixel_format == V4L2_PIX_FMT_SRGGB8 ||
+                params.pixel_format == V4L2_PIX_FMT_SBGGR8 ||
+                params.pixel_format == V4L2_PIX_FMT_SGBRG8 ||
+                params.pixel_format == V4L2_PIX_FMT_SGRBG8)
+            {
+                EXPECT_EQ((size_t)esz.area(), img.total());
+            }
+            else
+            {
+                EXPECT_EQ(esz, img.size());
+            }
             EXPECT_EQ(params.channels, img.channels());
             EXPECT_EQ(params.depth, img.depth());
         }

Currently V4L2 returns a single line in Bayer formats.

…_SRGGB8, V4L2_PIX_FMT_SBGGR8, V4L2_PIX_FMT_SGBRG8, V4L2_PIX_FMT_SGRBG8 options. Update modules/videoio/test/test_v4l2.cpp test file.
@catree catree force-pushed the feat_V4L2_PIX_FMT_SGRBG8_use_cvtColor branch from c10d0c2 to d81cd13 Compare April 3, 2024 11:48
Copy link
Copy Markdown
Contributor

@asmorkalov asmorkalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@asmorkalov asmorkalov merged commit 2b1c8aa into opencv:4.x Apr 4, 2024
@asmorkalov asmorkalov mentioned this pull request Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: videoio cleanup Code cleanup (e.g, drop legacy C-API, legacy unmaintained code)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants