videoio: Add raw encoded video stream encapsulation to `cv::VideoWriter` with `CAP_FFMPEG` by cudawarped · Pull Request #24363 · opencv/opencv

cudawarped · 2023-10-05T13:33:39Z

Allow raw encoded video streams (e.g. h264[5]) to be encapsulated by cv::VideoWriter to video containers (e.g. mp4/mkv).

Operates in a similar way to #15290 where encapsulation is enabled by setting the VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO flag when constructing cv::VideoWriter e.g.

VideoWriter container(fileNameOut, api, fourcc, fps, { width, height }, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1 });

and each raw encoded frame is passed as single row of a CV_8U cv::Mat.

The main reason for this PR is to allow cudacodec::VideoWriter to output its encoded streams to a suitable container, see opencv/opencv_contrib#3569.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

cudawarped · 2023-10-05T16:58:15Z

@asmorkalov Thanks for running the CI, unfortunately I left in one of my own tests

[ RUN ] videoio_encapsulate.write/6, where GetParam() = ("video/big_buck_bunny.vob", "mp4", 125, 125)
unknown file: Failure
C++ exception with description "OpenCV(4.8.0-dev) /home/ci/opencv/modules/ts/src/ts.cpp:1071: error: (-2:Unspecified error) OpenCV tests: Can't find required data file: video/big_buck_bunny.vob in function 'findData'
" thrown in the test body.
[ FAILED ] videoio_encapsulate.write/6, where GetParam() = ("video/big_buck_bunny.vob", "mp4", 125, 125) (0 ms)

I've removed the test, so everything should pass now.

asmorkalov · 2023-10-18T12:53:39Z

n file included from /build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg.cpp:50:0:
/build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp: In function 'int icv_av_encapsulate_video_FFMPEG(AVFormatContext*, AVStream*, AVCodecContext*, uint8_t*, int, int, bool)':
/build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:2337:37: error: 'av_packet_alloc' was not declared in this scope
     AVPacket* pkt = av_packet_alloc();
                                     ^
/build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:2345:24: error: 'av_packet_free' was not declared in this scope
     av_packet_free(&pkt);
                        ^
/build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp: In member function 'bool CvVideoWriter_FFMPEG::open(const char*, int, double, int, int, const cv::VideoWriterParameters&)':
/build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:3086:9: error: 'AVHWDeviceType' was not declared in this scope
         AVHWDeviceType hw_type = AV_HWDEVICE_TYPE_NONE;
         ^
/build/precommit_linux64/4.x/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:3085:23: warning: unused variable 'hw_format' [-Wunused-variable]
         AVPixelFormat hw_format = AV_PIX_FMT_NONE;
                       ^
modules/videoio/CMakeFiles/opencv_videoio.dir/build.make:302: recipe for target 'modules/videoio/CMakeFiles/opencv_videoio.dir/src/cap_ffmpeg.cpp.o' failed
make[2]: *** [modules/videoio/CMakeFiles/opencv_videoio.dir/src/cap_ffmpeg.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....

modules/videoio/src/cap_ffmpeg_impl.hpp

modules/videoio/include/opencv2/videoio.hpp

asmorkalov · 2023-10-18T13:42:33Z

Looks like av_packet_alloc and other mentioned functions are not implemented in FFmpeg n2.8.17 provided with Ubuntu 16.04. I understand, that it's old, but we need to able to build it even with limited support.

cudawarped · 2023-10-19T18:36:56Z

[ RUN ] videoio_encapsulate.write/17, where GetParam() = ("video/sample_322x242_15frames.yuv420p.libx265.mp4", "mp4", 15, 15)
OpenCV: FFMPEG: tag 0x63766568/'hevc' is not supported with codec id 173 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x31766568/'hev1'
Checking encapsulated video container: /tmp/__opencv_temp.DBKT5f.test_encapsulated_stream.mp4
Error: The action has timed out.

This is the same test which timed out last time, I don't have access to macOS-ARM64 to debug this, should I just disable that test to see if it passes?

asmorkalov · 2023-10-20T13:42:59Z

I get strange test failures locally:

[  FAILED  ] 6 tests, listed below:
[  FAILED  ] videoio_encapsulate.write/0, where GetParam() = ("video/big_buck_bunny.h264", "mp4", 125, 125)
[  FAILED  ] videoio_encapsulate.write/1, where GetParam() = ("video/big_buck_bunny.h265", "mp4", 125, 125)
[  FAILED  ] videoio_encapsulate.write/16, where GetParam() = ("video/sample_322x242_15frames.yuv420p.libx264.mp4", "mp4", 15, 15)
[  FAILED  ] videoio_encapsulate.write/17, where GetParam() = ("video/sample_322x242_15frames.yuv420p.libx265.mp4", "mp4", 15, 15)
[  FAILED  ] videoio_encapsulate.write/18, where GetParam() = ("video/sample_322x242_15frames.yuv420p.libx264.mp4", ".mp4", 15, 15)
[  FAILED  ] videoio_encapsulate.write/22, where GetParam() = ("../cv/video/768x576.avi", "mp4", 15, 16)

Example:

[ RUN      ] videoio_encapsulate.write/0, where GetParam() = ("video/big_buck_bunny.h264", "mp4", 125, 125)
[ INFO:0@0.135] global cap_ffmpeg_impl.hpp:1047 open VIDEOIO/FFMPEG: enabled demuxer only mode: '/home/alexander/opencv_extra/testdata/highgui/video/big_buck_bunny.h264'
[DEBUG:0@0.142] global cap_ffmpeg_impl.hpp:2870 open Selected pixel format: bgr24
OpenCV: FFMPEG: tag 0x34363268/'h264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x31637661/'avc1'
Checking encapsulated video container: /tmp/__opencv_temp.G7BrNb.test_encapsulated_stream.mp4
[DEBUG:0@0.149] global cap_ffmpeg_impl.hpp:1203 open FFMPEG: stream[0] is video stream with codecID=28 width=672 height=384
[DEBUG:0@0.155] global cap_ffmpeg_impl.hpp:1203 open FFMPEG: stream[0] is video stream with codecID=28 width=672 height=384
[ INFO:0@0.155] global cap_ffmpeg_impl.hpp:1047 open VIDEOIO/FFMPEG: enabled demuxer only mode: '/tmp/__opencv_temp.G7BrNb.test_encapsulated_stream.mp4'
[DEBUG:0@0.168] global cap_ffmpeg_impl.hpp:1638 retrieveFrame Input picture format: yuv420p
[DEBUG:0@0.176] global cap_ffmpeg_impl.hpp:1638 retrieveFrame Input picture format: yuv420p
/home/alexander/Projects/OpenCV/opencv-master/modules/videoio/test/test_ffmpeg.cpp:314: Failure
Expected equality of these values:
  keyFrameReference
    Which is: true
  keyFrameActual
    Which is: false
[  FAILED  ] videoio_encapsulate.write/0, where GetParam() = ("video/big_buck_bunny.h264", "mp4", 125, 125) (48 ms)
[----------] 1 test from videoio_encapsulate (48 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (48 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] videoio_encapsulate.write/0, where GetParam() = ("video/big_buck_bunny.h264", "mp4", 125, 125)

OS: Ubuntu 18.04 with default FFmpeg instance.

cudawarped · 2023-10-20T14:33:42Z

I knew this PR was a bad idea, so it works on FFMpeg 2.8.17 and 4.2.4 but not 3.4.8. I'll take a look.

cudawarped · 2023-10-22T05:09:27Z

@asmorkalov It looks like the problem is not that the keyframe is not marked when the stream is encapsulated but that FFMpeg discards that information when the stream is read back and filtered.

In FFMpeg >= 3.1 av_bitstream_filter_filter takes keyframe as its last argument but doesn't use it. I haven't looked through all of the internals to check what is going on and to see if the packet.flags argument is used somewhere but incorrectly however the end result is that if packet.flags & AV_PKT_FLAG_KEY == 1, packet_filtered.flags & AV_PKT_FLAG_KEY != 1.

I've introduced a workaround which fixes the issue on Ubuntu 20.04 with OpencV built against the lattest version of FFMpeg 3.4 (3.4.13) and would appreciate it if you could test it again on your Ubuntu 18.04 instance.

…r with CAP_FFMEG

asmorkalov · 2023-10-23T16:58:29Z

@cudawarped Thanks a lot for the effort. I tried Ubuntu 16.04, 18.04, 22.04. The last 2 works great! Ubuntu 16.04 triggers some errors with h264 and h265 codecs in the same way as on Buildbot. I'll try to debug it tomorrow. Error message example:

Note: Google Test filter = videoio_encapsulate.write/0
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from videoio_encapsulate
[ RUN      ] videoio_encapsulate.write/0, where GetParam() = ("video/big_buck_bunny.h264", "mp4", 125, 125)
OpenCV: FFMPEG: tag 0x34363268/'h264' is not supported with codec id 28 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x00000021/'!???'
Checking encapsulated video container: /tmp/__opencv_temp.ri43re.test_encapsulated_stream.mp4
[NULL @ 0x311770] Packet header is not contained in global extradata, corrupted stream or invalid MP4/AVCC bitstream
/home/ubuntu/Projects/opencv/modules/videoio/test/test_ffmpeg.cpp:311: Failure
Value of: capActualRaw.grab()
  Actual: false
Expected: true
[  FAILED  ] videoio_encapsulate.write/0, where GetParam() = ("video/big_buck_bunny.h264", "mp4", 125, 125) (212 ms)

asmorkalov · 2023-10-24T15:20:05Z

@cudawarped I tuned test parameters a bit to satisfy old FFmpeg versions. Looks like they support not all combinations of codec-container. Please take a look.

cudawarped · 2023-10-24T16:44:00Z

@asmorkalov Nice fix, working on my Ubuntu 20.04 with FFMpeg 2.8.17 and its passing on IOS!

I was also thinking to reduce the number of frames in the following to 10 as there is only one key frame so it doesn't make any sense to read all 125.

i.e. From

videoio_encapsulate_params_t("video/big_buck_bunny.h264", "avi", 125, 125),
videoio_encapsulate_params_t("video/big_buck_bunny.h265", "mp4", 125, 125),
videoio_encapsulate_params_t("video/big_buck_bunny.avi", "mp4", 125, 125),

to

videoio_encapsulate_params_t("video/big_buck_bunny.h264", "avi", 10, 10),
videoio_encapsulate_params_t("video/big_buck_bunny.h265", "mp4", 10, 10),
videoio_encapsulate_params_t("video/big_buck_bunny.avi", "mp4", 10, 10),

Do you want to do that or should I?

asmorkalov

👍

…ream_encapsulation videoio: Add raw encoded video stream muxing to cv::VideoWriter with CAP_FFMPEG opencv#24363 Allow raw encoded video streams (e.g. h264[5]) to be encapsulated by `cv::VideoWriter` to video containers (e.g. mp4/mkv). Operates in a similar way to opencv#15290 where encapsulation is enabled by setting the `VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO` flag when constructing `cv::VideoWriter` e.g. ``` VideoWriter container(fileNameOut, api, fourcc, fps, { width, height }, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1 }); ``` and each raw encoded frame is passed as single row of a `CV_8U` `cv::Mat`. The main reason for this PR is to allow `cudacodec::VideoWriter` to output its encoded streams to a suitable container, see opencv/opencv_contrib#3569. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

cudawarped mentioned this pull request Oct 5, 2023

cudacodec::VideoWriter add container output opencv/opencv_contrib#3569

Merged

6 tasks

asmorkalov added the category: videoio label Oct 5, 2023

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch from cda7666 to 16c58e5 Compare October 5, 2023 16:49

cudawarped changed the title ~~videoio: Add raw encoded video stream encapsulation to cv::VideoWriter with CAP_FFMEG~~ videoio: Add raw encoded video stream encapsulation to cv::VideoWriter with CAP_FFMPEG Oct 6, 2023

asmorkalov self-requested a review October 18, 2023 13:01

asmorkalov self-assigned this Oct 18, 2023

asmorkalov added this to the 4.9.0 milestone Oct 18, 2023

asmorkalov reviewed Oct 18, 2023

View reviewed changes

modules/videoio/src/cap_ffmpeg_impl.hpp Outdated Show resolved Hide resolved

modules/videoio/include/opencv2/videoio.hpp Outdated Show resolved Hide resolved

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch from 16c58e5 to b0f4231 Compare October 19, 2023 12:13

opencv-alalek added the feature label Oct 20, 2023

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch from b0f4231 to c03ae53 Compare October 22, 2023 04:29

videoio: Add raw encoded video stream encapsulation to cv::VideoWrite…

6ebc81e

…r with CAP_FFMEG

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch from c03ae53 to 6ebc81e Compare October 22, 2023 05:37

Tunned test parameters to handle old FFmpeg (Ubuntu 16.04).

558e8cb

asmorkalov added 2 commits October 25, 2023 11:49

Warnings fix on Windows.

1b5b4a0

More Windows warning fixes.

ddd4cc5

asmorkalov approved these changes Oct 25, 2023

View reviewed changes

asmorkalov merged commit 38bc519 into opencv:4.x Oct 25, 2023

asmorkalov mentioned this pull request Nov 3, 2023

(5.x) Merge 4.x #24486

Merged

asmorkalov mentioned this pull request Jul 19, 2024

videoio: fix cv::VideoWriter with FFmpeg encapsulation timestamps #25874

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

videoio: Add raw encoded video stream encapsulation to `cv::VideoWriter` with `CAP_FFMPEG`#24363

videoio: Add raw encoded video stream encapsulation to `cv::VideoWriter` with `CAP_FFMPEG`#24363
asmorkalov merged 4 commits intoopencv:4.xfrom
cudawarped:videoio_ffmpeg_add_stream_encapsulation

cudawarped commented Oct 5, 2023 •

edited

Loading

Uh oh!

cudawarped commented Oct 5, 2023

Uh oh!

asmorkalov commented Oct 18, 2023

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Oct 18, 2023

Uh oh!

cudawarped commented Oct 19, 2023

Uh oh!

asmorkalov commented Oct 20, 2023 •

edited

Loading

Uh oh!

cudawarped commented Oct 20, 2023

Uh oh!

cudawarped commented Oct 22, 2023

Uh oh!

asmorkalov commented Oct 23, 2023

Uh oh!

asmorkalov commented Oct 24, 2023

Uh oh!

cudawarped commented Oct 24, 2023 •

edited

Loading

Uh oh!

asmorkalov left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cudawarped commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

cudawarped commented Oct 5, 2023

Uh oh!

asmorkalov commented Oct 18, 2023

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Oct 18, 2023

Uh oh!

cudawarped commented Oct 19, 2023

Uh oh!

asmorkalov commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cudawarped commented Oct 20, 2023

Uh oh!

cudawarped commented Oct 22, 2023

Uh oh!

asmorkalov commented Oct 23, 2023

Uh oh!

asmorkalov commented Oct 24, 2023

Uh oh!

cudawarped commented Oct 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cudawarped commented Oct 5, 2023 •

edited

Loading

asmorkalov commented Oct 20, 2023 •

edited

Loading

cudawarped commented Oct 24, 2023 •

edited

Loading