Skip to content

add Y16/GRAY16_LE support to gstreamer#14035

Closed
jveitchmichaelis wants to merge 1 commit intoopencv:3.4from
jveitchmichaelis:y16_gstreamer
Closed

add Y16/GRAY16_LE support to gstreamer#14035
jveitchmichaelis wants to merge 1 commit intoopencv:3.4from
jveitchmichaelis:y16_gstreamer

Conversation

@jveitchmichaelis
Copy link
Copy Markdown
Contributor

@jveitchmichaelis jveitchmichaelis commented Mar 12, 2019

This pullrequest changes

  • Adds Y16 (GRAY16_LE) support to Gstreamer.
  • Adds support for CAP_PROP_CONVERT_RGB
  • Adds a depth parameter to the gstreamer capture class
  • Images are returned as CV_8UC3 by default, unless CAP_PROP_CONVERT_RGB is set to false.

Tested with a FLIR Boson

@jveitchmichaelis jveitchmichaelis force-pushed the y16_gstreamer branch 2 times, most recently from 87e924b to 1003f66 Compare March 12, 2019 12:53
@jveitchmichaelis
Copy link
Copy Markdown
Contributor Author

Fixed trailing whitespace

@jveitchmichaelis jveitchmichaelis force-pushed the y16_gstreamer branch 2 times, most recently from 9f36a8a to ce389e4 Compare March 12, 2019 12:58
@mshabunin
Copy link
Copy Markdown
Contributor

@jveitchmichaelis , could you please add test case here: https://github.com/opencv/opencv/blob/master/modules/videoio/test/test_gstreamer.cpp

Is this patch applicable to 3.4 branch too? If yes, please rebase your branch there and change pull request target to 3.4 branch. We will merge it to master later (see https://github.com/opencv/opencv/wiki/Branches).

@jveitchmichaelis jveitchmichaelis changed the base branch from master to 3.4 March 13, 2019 19:21
@jveitchmichaelis
Copy link
Copy Markdown
Contributor Author

jveitchmichaelis commented Mar 13, 2019

@mshabunin No problem. I've rebased to 3.4, I'll add some tests too.

The main differences I can see were some checks for a particular version of gstreamer that are no longer in master. I added support for writing 16-bit frames with Gstreamer, but perhaps it makes sense to split that into another commit and leave this one only for capture.

At a minimum I can add a testcase like:

make_tuple("video/x-raw, format=GRAY16_LE", Size(640, 480), Size(640, 480), COLOR_GRAY2RGB),

Would that be sufficient?

@mshabunin
Copy link
Copy Markdown
Contributor

I added support for writing 16-bit frames with Gstreamer, but perhaps it makes sense to split that into another commit and leave this one only for capture.

I agree.

At a minimum I can add a testcase like:
...
Would that be sufficient?

Yes, at minimum. It would be better to test handling of CV_CAP_PROP_CONVERT_RGB property too and verify actual 16-bit image capturing. But we can add it in later PRs.

@jveitchmichaelis
Copy link
Copy Markdown
Contributor Author

jveitchmichaelis commented Mar 17, 2019

@mshabunin I've added two test cases. I also limited the scope of the PR to capture only, as suggested. There is:

  • A test case to check the default behaviour for CONVERT_RGB
  • An additional case for the existing suite, however..

One issue that I've noticed is that CAP_GSTREAMER doesn't follow the convention that capture pipelines should return 3-channel 8-bit images by default i.e. convert_rgb is ignored. Most of the outputs here (e.g. GRAY8, and all the YUV variants) return a single channel image and rely on the user to convert to colour. The existing test cases also (incorrectly) enforces this behaviour.

I've raised that in #14084

Seems to be an issue with the linux debug CI too: https://pullrequest.opencv.org/buildbot/builders/precommit_linux64_no_opt/builds/19060

Copy link
Copy Markdown
Contributor

@mshabunin mshabunin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jveitchmichaelis , sorry for delay.

I think we should remove CONVERT_RGB property from this PR. GStreamer backend is special because it allows to run manual pipelines, where user can specify output format. We have two modes: automatic pipeline - always return 8UC3 or 8UC1 and manual pipeline - return raw format requested by the user. So we just need to add 16-bit format to the list of supported formats for manual pipelines and test it. The test can be modified, so that it would call something like convertScaleAbs before doing cvtColor(GRAY2BGR).

@joshdoe
Copy link
Copy Markdown
Contributor

joshdoe commented Apr 24, 2019

I added support for writing 16-bit frames with Gstreamer, but perhaps it makes sense to split that into another commit and leave this one only for capture.

@jveitchmichaelis, we'd be happy to test a patch for writing, please create a PR when you get a chance. We looked at doing it, but it's not obvious how we could automatically determine a pipeline is Y16, seems like we'd need a property.

@jveitchmichaelis
Copy link
Copy Markdown
Contributor Author

jveitchmichaelis commented Apr 26, 2019

@mshabunin OK, I'll take a look at this again. I've not had a chance recently. So I'll change it so that 16 bit is only supported for manual pipelines.

@joshdoe sure, will do! It will probably be doable based on the output codec. Relatively few formats actually support 16 bit output per channel, so you could infer on that plus the pixel format I think.

@joshdoe
Copy link
Copy Markdown
Contributor

joshdoe commented Apr 26, 2019

@joshdoe sure, will do! It will probably be doable based on the output codec. Relatively few formats actually support 16 bit output per channel, so you could infer on that plus the pixel format I think.

@jveitchmichaelis Great, thanks. I recently added Y16 support to the GStreamer DirectShow source because I was testing a FLIR Boson thermal camera, but in this case we're using a Camera Link source (https://github.com/joshdoe/gst-plugins-vision).

@joshdoe
Copy link
Copy Markdown
Contributor

joshdoe commented Oct 7, 2020

I've modified this to apply against master, as I couldn't easily get the 3.4 branch built. I removed the CONVERT_RGB bits as suggested by @mshabunin , and changed the tests a bit to check depth on all formats, and convert 16->8 for GRAY16_LE. My branch is joshdoe/opencv/y16_gstreamer_master. I also rebased this to 3.4, but haven't been able to test it: joshdoe/opencv/y16_gstreamer_3.4.

Note the authorship changed to me, but I added Co-author as @jveitchmichaelis , I'm fine if this is flipped.

@asmorkalov
Copy link
Copy Markdown
Contributor

@jveitchmichaelis The PR diverged with 3.4. Could you rebase and fix conflicts.
@mshabunin What is the PR status? Are there open items? Could we merge it after rebase and conflicts fix?

@mshabunin
Copy link
Copy Markdown
Contributor

@asmorkalov , I think #18694 is more compact and implements the same feature, so we can close this one. #18535 implements Y16 support in VideoWriter.

@joshdoe
Copy link
Copy Markdown
Contributor

joshdoe commented Nov 24, 2020

I'll second the implementation in #18694 as it's cleaner, though I tested writing (from #18535) and reading for both of these implementations and they passed.

@jveitchmichaelis
Copy link
Copy Markdown
Contributor Author

Apologies, this feel through the cracks - feel free to go with whichever implementation you feel fits best.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants