cv::UMat output/input in VideoCapture/VideoWriter (data stays in GPU memory) by mikhail-nikolskiy · Pull Request #19755 · opencv/opencv

mikhail-nikolskiy · 2021-03-20T01:13:30Z

The pull request utilizes HW acceleration added recently to FFMPEG backend of videoio module, and core APIs for interoperability with DirectX and VAAPI.
Interoperability requires OpenCL context created with DirectX/VAAPI device passed at context init time. So in case of HW decode/encode and cv::UMat, data can stay in GPU memory if VideoCapture/VideoWriter open() called before any OpenCL-related functions so OpenCL context is empty at open() time, or OpenCL context explicitly initialized in interop mode before open() via calling functions cv::directx::ocl::initializeContextFromD3D*Device/cv::va_intel::ocl::initializeContextFromVA. Otherwise data copied through system memory GPU->CPU->GPU.

OpenCL context before open()	parameter PROP_HW_DEVICE	OpenCL context after open()	data stays in GPU, if cv::UMat and HW mode
empty	supported	created via initializeContextFrom*, if HW mode	Yes
created via initializeContextFrom*	ignored, with log message if specified	unchanged	Yes
created, but not via initializeContextFrom*	supported	unchanged	No

For example, if app creates VideoCapture then VideoWriter instances with parameters PROP_HW_ACCELERATION and PROP_HW_DEVICE before OpenCL calls, the first-opened VideoCapture will try open GPU device specified in PROP_HW_DEVICE, the second-opened VideoWriter will reuse media/OpenCL device instance previously created by VideoCapture.

The pull request also introduces new functions in ocl::Context to unify capabilities of attaching external context to OpenCL context, as it used now in three places directx.cpp (core), va_intel.cpp (core), cap_ffmpeg_hw.hpp (videoio).

relates #19676

TODO (alalek):

move out HAVE_MFX of cvconfig.h
properly configure HAVE_MFX / HAVE_OPENCL for FFmpeg plugin (with include directories)
... HAVE_VA / HAVE_VA_INTEL / etc for FFmpeg plugin (with include directories)
(backlog, out of scope of this PR) videoio plugins API - retrieve OpenCL buffers (currently it is limited to cv::Mat only)
(backlog, out of scope of this PR) FFmpeg for Windows: build wrapper with enable OpenCL support and share OpenCL context
try to reuse old prebuilt videoio wrappers (like FFmpeg for Windows) in OpenCV tests (some features may missing)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

build_contrib:Custom=OFF
build_contrib:Custom Win=OFF

build_shared:Custom=OFF
build_examples:Custom=OFF

build_image:Linux AVX2=ubuntu:20.04

Xbuild_image:Win64=msvs2019

force_builders=Custom,Custom Win,Linux AVX2
Xbuild_image:Custom=centos:7
Xbuild_image:Custom=ubuntu:20.04
build_image:Custom=gstreamer:16.04
buildworker:Custom=linux-1,linux-4,linux-6
test_opencl:Custom=ON

build_image:Custom Win=ffmpeg
Xbuild_image:Custom Win=ffmpeg-plugin
buildworker:Custom Win=windows-2
test_modules:Custom Win=videoio
test_opencl:Custom Win=ON

mikhail-nikolskiy · 2021-03-26T03:43:47Z

UMat output/input works in my local testing on VAAPI decoders/encoders (Ubuntu 20.04), D3D11 decoders (Windows 10), MFX decoders/encoders (Ubuntu 20.04 and Windows 10).
If GPU memory interop not possible by any reason (including below), UMat output/input works via CPU memory.
Known limitations for GPU memory interop:

GPU memory interop not supported in videoio plugin mode
Test system creates OpenCL context on initialization, so video acceleration tests can't trigger GPU memory interop mode unless insert context deletion 'ocl::OpenCLExecutionContext::getCurrent().release()' at start of video acceleration tests
If MFX acceleration, GPU memory interop requires HAVE_MFX enabled. The actual dependency is only on MFX headers, no linkage dependency.
If MFX acceleration, GPU memory interop requires custom FFMPEG build with patches for MFX+D3D11 support developed by @galinart, these patches not upstreamed to FFMPEG master yet.

mikhail-nikolskiy · 2021-03-26T03:46:38Z

modules/core/include/opencv2/core/ocl.hpp


 CV_EXPORTS_W bool haveOpenCL();
-CV_EXPORTS_W bool useOpenCL();
+CV_EXPORTS_W bool useOpenCL(bool init = true); // TODO: can we change function signature?


@alalek Should be another function, as we can't change signature of existent function?

mikhail-nikolskiy · 2021-03-26T04:00:37Z

@asmorkalov, this PR has 4 failing tests for vp9 decoding on Windows 10, is it same system setup and probably same reason as issue 19694?

asmorkalov · 2021-03-29T07:41:48Z

@mikhail-nikolskiy Yes.

asmorkalov · 2021-03-29T07:45:14Z

jenkins cn please retry a build

asenyaev · 2021-04-05T20:38:24Z

jenkins cn please retry a build

alalek

Thank you for great contribution!

I'm sorry for huge review delay (we were busy with release activities).
This is the first review iteration.

alalek · 2021-04-06T21:59:52Z

modules/videoio/src/cap_ffmpeg_hw.hpp

+            CV_LOG_INFO(NULL, "FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for " << hw_child_name << " on device " << device_name);
+            // if OpenCL context not created yet, create it with binding to video acceleration context
+            if (ocl::useOpenCL(false)) {
+                if (ocl_context.empty()) {


ocl::useOpenCL(false)

I believe it is enough to check ocl_context.empty() only.

So adding new "useOpenCL" behavior is not necessary.

Current opencv_test_videoio tests stuck with these messages (due to pre-initialized OpenCL device):

[ RUN ] videoio/videocapture_acceleration.read/55, where GetParam() = (sample_322x242_15frames.yuv420p.libx264.mp4, FFMPEG, VAAPI, true) ... [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (268) hw_check_device FFMPEG: Using vaapi video acceleration on device: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 () [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (559) hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for vaapi on device 'default' [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (571) hw_create_device FFMPEG: Can't bind vaapi video acceleration context to already created OpenCL context

BTW, I would suggest to add explicit "VideoCapture" open parameter which forces creation of new of OpenCL context.

alalek · 2021-04-08T03:10:29Z

modules/videoio/src/cap_ffmpeg_hw.hpp

+    return AV_HWDEVICE_TYPE_NONE;
+}
+
+static void hw_init_opencl(AVBufferRef* ctx) {


There is an regression for case with disabled OpenCL subsystem in OpenCV through OPENCV_OPENCL_RUNTIME=disabled. Both Mat / UMat test cases don't work (skipped) with messages:

[OPENCV:FFMPEG:40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 20.4.5 (74e2f111). [OPENCV:FFMPEG:40] Driver not found in known nonstandard list, using standard behaviour. [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (268) hw_check_device FFMPEG: Using vaapi video acceleration on device: Intel iHD driver for Intel(R) Gen Graphics - 20.4.5 (74e2f111) [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (559) hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for vaapi on device 'default' [ERROR:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap.cpp (160) open VIDEOIO(FFMPEG): raised OpenCV exception: OpenCV(4.5.2-dev) /home/alalek/projects/opencv/dev/modules/core/src/opencl/runtime/opencl_core.cpp:378: error: (-220:Unknown error code -220) OpenCL function is not available: [clGetPlatformIDs] in function 'opencl_check_fn' [ SKIP ] FFMPEG VideoCapture on sample_322x242_15frames.yuv420p.libx264.mp4 not supported with HW acceleration, skipping

alalek · 2021-04-08T03:17:59Z

modules/core/include/opencv2/core/ocl.hpp

    void* ptr() const;

+    /** @returns Property specified on context creation */
+    void* getProperty(long propertyId) const;


As we want to work with OpenCL enums lets use more specific method name getOpenCLContextProperty() and add note to doc string about returned values and passed parameter (extract properties through clGetContextInfo(CL_CONTEXT_PROPERTIES) call)

alalek · 2021-04-08T03:24:10Z

modules/core/src/ocl.cpp

+    if (CL_SUCCESS == clGetContextInfo(p->handle, CL_CONTEXT_PROPERTIES, 0, NULL, &size)) {
+        std::vector<cl_context_properties> prop(size / sizeof(cl_context_properties));
+        if (CL_SUCCESS == clGetContextInfo(p->handle, CL_CONTEXT_PROPERTIES, size, prop.data(), NULL)) {


Just a question: can we cache this properties array? or its contents can be changed in runtime

OpenCL doesn't allow to change properties (I would really like to have such ability - change media context property after context creation;)
But properties cached by OpenCL user-mode library, so these these two calls should work very fast.
Is it ok, or better cache on OpenCV side?

Thanks for information. Caching support here is low priority for now.

modules/core/include/opencv2/core/ocl.hpp

alalek · 2021-04-08T03:31:17Z

modules/core/include/opencv2/core/ocl.hpp


+    class CV_EXPORTS UserContext {
+    public:
+        virtual ~UserContext() {};


It makes sense to put destructor implementation in .cpp file.
Otherwise "full header" classes may lead to RTTI issues (e.g. with clang - it creates independent clones in each module)

alalek · 2021-04-08T03:47:15Z

cmake/templates/cvconfig.h.in

 #cmakedefine HAVE_VA_INTEL

+/* Intel MediaSDK/oneVPL */
+#cmakedefine HAVE_MFX


TODO(alalek): check how to move this to videoio plugins

alalek · 2021-04-08T03:50:00Z

modules/videoio/src/cap_ffmpeg_impl.hpp

+
+    // check that current OpenCL context initilized on same media device as codec
+    //if (!hw_check_opencl_context(video_st->codec->hw_device_ctx))
+    //    return false;


Perhaps robust check should check UMat's OpenCL context.

It would require code like below?

CV_Assert(u.u && u.u->currAllocator == ocl::getOpenCLAllocator()); CV_Assert(std::static_pointer_cast<ocl::Context>(u.u->allocatorContext)->ptr() == ocl_context.getContext().ptr());

I believe SetKernelArgument will throw exception if wrong context, and this code under try/catch - is it ok or better check context?

Not sure that OpenCL CTS has checks for "wrong context" cases.

I would prefer to raise OpenCV exception instead of crash in OpenCL runtime internals.

asenyaev · 2021-04-08T12:33:41Z

jenkins cn please retry a build

mikhail-nikolskiy · 2021-04-09T04:40:28Z

Thanks for review!
I pushed commit addressing review comments.

Regarding "explicit VideoCapture open parameter which forces creation of new of OpenCL context".
I added such parameter preliminary named CAP_PROP_HW_DEVICE_OPENCL, and added it to tests.
Such draft naming because it could be used potentially for more functionality - force OpenCL context creation and select GPU device via value of cl_device_id type (works with all acceleration types including VIDEO_ACCELERATION_ANY), as alternative way of device selection via CAP_PROP_HW_DEVICE with index in acceleration specific enumeration.
So that user can do

    cl_device_id clDeviceId = ... // select device by cv::ocl calls or native OpenCL calls
    VideoCapture capture(infile, backend, {
            CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY,
            CAP_PROP_HW_DEVICE_OPENCL, clDeviceId
    });

or

    VideoCapture capture(infile, backend, {
            CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_D3D11,
            CAP_PROP_HW_DEVICE, 1, // D3D11 device#1
            CAP_PROP_HW_DEVICE_OPENCL, NULL // force OpenCL context creation on device#1
    });

Is it beneficial?
It should be possible via device UUID comparison, but the problem that cl_device_id is pointer and properties currently don't support pointers as value.

alalek · 2021-04-09T10:12:32Z

wants to merge 155 commits

js: support setLogLevel() / getLogLevel() calls
Merge branch 'ffmpeg-umat' of https://github.com/mikhail-nikolskiy/op

Looks like "git merge" has been done in the wrong direction.
Need to merge "upstream" (external) commits, not yours.

alalek · 2021-04-09T10:19:17Z

CAP_PROP_HW_DEVICE_OPENCL

Looks like using some handles here is over-complicated for users.

I would prefer to have ON/OFF property:

CAP_PROP_HW_ACCELERATION_USE_OPENCL=OFF (default)
- check and re-use existed OpenCL context if context is compatible.
- disable OpenCL buffers if context is not compatible.
CAP_PROP_HW_ACCELERATION_USE_OPENCL=ON
- check and re-use existed OpenCL context if context is compatible.
- create new "Media" context if current context is not compatible (replaces threads's OpenCLExecutionContext).

mikhail-nikolskiy · 2021-04-10T01:31:26Z

Looks like using some handles here is over-complicated for users.

I renamed property to CAP_PROP_HW_ACCELERATION_USE_OPENCL with zero/non-zero value, and fixed merge problem.

alalek · 2021-04-12T19:54:58Z

Excluded "merge" and "revert" commits from the PR (no changes in the code).
Please fetch changes from the source branch of your GitHub fork (without merge or git pull).

…L GPU+CPU platform

mikhail-nikolskiy · 2021-04-13T01:12:25Z

Thanks for removing mess with revert commits!
Added fix today for DX11 problem reported by @rboriso1 - OpenCL platform with GPU+CPU devices reports only extensions supported by both GPU and CPU devices, so doesn't report DX11 interop extension although it supported by GPU. The change is to try load extension function without checking extension itself, as done already in VAAPI/DX9/DX10.

…ntext from OpenCL context

mikhail-nikolskiy · 2021-04-14T04:11:26Z

On Windows 10 builder I saw test failures on test cases with use_umat=true parameter, but can't find any log messages from test execution (exceptions, errors, etc). Is it possible to get debug log? It would help a lot as can't reproduce on local iGPU and dGPU devices.

alalek · 2021-04-14T15:43:10Z

On Windows 10 builder I saw test failures on test cases with use_umat=true parameter

Win64 / Win64 OpenCL builders uses prebuilt FFmpeg wrapper. It doesn't contain changes from this patch.
I need to think how to properly disable these test cases in case of old FFmpeg wrapper.

alalek

Thank you for contribution!
This patch works well in these configurations on my side (validated):

Linux + VAAPI + Intel iGPU
Windows (with FFmpeg build) + D3D11 + Intel iGPU

Remaining CI "errors" are specific to OpenCV / build configurations. I will take a look how to resolve them (added list to PR's description)

modules/core/src/directx.cpp

dmatveev · 2021-04-28T17:47:44Z

That's interesting stuff for sure!

@mikhail-nikolskiy would you mind adding support for cv::RMat & cv::MediaFrame as well? Those data structures are especially designed to handle the remote memory with custom adaptors inside (e.g. this way we to zero-copy for GStreamer buffers in the G-API pipeline), so may be worth adding too. The latter cv::MediaFrame structure brings more structured handling for the media formats like NV12 (and other YUV variants can be added there, too).

mikhail-nikolskiy · 2021-04-29T22:13:22Z

@mikhail-nikolskiy would you mind adding support for cv::RMat & cv::MediaFrame as well?

I checked RMat::access, it seems to be functionally equal to UMat::getMat, so wrapper from UMat to RMat could be implemented in generic way, not specifically in videoio module?
(Dmitry, your question actually triggers important topic around MatAllocator/Adaptor concepts, but probably better discuss separately)

- plugins don't support OpenCL/UMat yet - change hanling of *_USE_OPENCL flag

alalek · 2021-05-03T09:59:23Z

modules/videoio/src/cap_ffmpeg_hw.hpp

    case VIDEO_ACCELERATION_D3D11: return "";
    case VIDEO_ACCELERATION_VAAPI: return "vaapi.iHD";
-    case VIDEO_ACCELERATION_MFX: return "";
+    case VIDEO_ACCELERATION_MFX: return "qsv.iHD";


There is a regression on Linux machine for the test sample_322x242_15frames.yuv420p.libvpx-vp9.mp4, FFMPEG, MFX, false

Test logs are below (OPENCV_FFMPEG_DEBUG=1 OPENCV_LOG_LEVEL=DEBUG):

With patch

[ RUN ] videoio/videocapture_acceleration.read/84, where GetParam() = (sample_322x242_15frames.yuv420p.libvpx-vp9.mp4, FFMPEG, MFX, false) [OPENCV:FFMPEG:48] Opening '/home/alalek/projects/opencv/extra/testdata/highgui/video/sample_322x242_15frames.yuv420p.libvpx-vp9.mp4' for reading [OPENCV:FFMPEG:48] Setting default whitelist 'file,crypto,data' [OPENCV:FFMPEG:56] Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:2048 [OPENCV:FFMPEG:56] Probing mp3 score:1 size:2048 [OPENCV:FFMPEG:48] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100 [OPENCV:FFMPEG:56] type:'ftyp' parent:'root' sz: 28 8 31057 [OPENCV:FFMPEG:48] ISO: File Type Major Brand: isom [OPENCV:FFMPEG:56] type:'free' parent:'root' sz: 8 36 31057 [OPENCV:FFMPEG:56] type:'mdat' parent:'root' sz: 30210 44 31057 [OPENCV:FFMPEG:56] type:'moov' parent:'root' sz: 811 30254 31057 [OPENCV:FFMPEG:56] type:'mvhd' parent:'moov' sz: 108 8 803 [OPENCV:FFMPEG:56] time scale = 1000 [OPENCV:FFMPEG:56] type:'trak' parent:'moov' sz: 597 116 803 [OPENCV:FFMPEG:56] type:'tkhd' parent:'trak' sz: 92 8 589 [OPENCV:FFMPEG:56] type:'edts' parent:'trak' sz: 36 100 589 [OPENCV:FFMPEG:56] type:'elst' parent:'edts' sz: 28 8 28 [OPENCV:FFMPEG:56] track[0].edit_count = 1 [OPENCV:FFMPEG:56] duration=600 time=0 rate=1.000000 [OPENCV:FFMPEG:56] type:'mdia' parent:'trak' sz: 461 136 589 [OPENCV:FFMPEG:56] type:'mdhd' parent:'mdia' sz: 32 8 453 [OPENCV:FFMPEG:56] type:'hdlr' parent:'mdia' sz: 45 40 453 [OPENCV:FFMPEG:56] ctype=[0][0][0][0] [OPENCV:FFMPEG:56] stype=vide [OPENCV:FFMPEG:56] type:'minf' parent:'mdia' sz: 376 85 453 [OPENCV:FFMPEG:56] type:'vmhd' parent:'minf' sz: 20 8 368 [OPENCV:FFMPEG:56] type:'dinf' parent:'minf' sz: 36 28 368 [OPENCV:FFMPEG:56] type:'dref' parent:'dinf' sz: 28 8 28 [OPENCV:FFMPEG:48] Unknown dref type 0x206c7275 size 12 [OPENCV:FFMPEG:56] type:'stbl' parent:'minf' sz: 312 64 368 [OPENCV:FFMPEG:56] type:'stsd' parent:'stbl' sz: 132 8 304 [OPENCV:FFMPEG:56] size=116 4CC=vp09 codec_type=0 [OPENCV:FFMPEG:56] type:'vpcC' parent:'stsd' sz: 20 8 30 [OPENCV:FFMPEG:56] type:'fiel' parent:'stsd' sz: 10 28 30 [OPENCV:FFMPEG:56] type:'stts' parent:'stbl' sz: 24 140 304 [OPENCV:FFMPEG:56] track[0].stts.entries = 1 [OPENCV:FFMPEG:56] sample_count=15, sample_duration=512 [OPENCV:FFMPEG:56] type:'stss' parent:'stbl' sz: 20 164 304 [OPENCV:FFMPEG:56] keyframe_count = 1 [OPENCV:FFMPEG:56] type:'stsc' parent:'stbl' sz: 28 184 304 [OPENCV:FFMPEG:56] track[0].stsc.entries = 1 [OPENCV:FFMPEG:56] type:'stsz' parent:'stbl' sz: 80 212 304 [OPENCV:FFMPEG:56] sample_size = 0 sample_count = 15 [OPENCV:FFMPEG:56] type:'stco' parent:'stbl' sz: 20 292 304 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 0, offset 2c, dts 0, size 5068, distance 0, keyframe 1 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 1, offset 13f8, dts 512, size 1308, distance 1, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 2, offset 1914, dts 1024, size 1265, distance 2, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 3, offset 1e05, dts 1536, size 1507, distance 3, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 4, offset 23e8, dts 2048, size 2089, distance 4, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 5, offset 2c11, dts 2560, size 1956, distance 5, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 6, offset 33b5, dts 3072, size 1795, distance 6, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 7, offset 3ab8, dts 3584, size 2204, distance 7, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 8, offset 4354, dts 4096, size 2267, distance 8, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 9, offset 4c2f, dts 4608, size 1315, distance 9, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 10, offset 5152, dts 5120, size 5251, distance 10, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 11, offset 65d5, dts 5632, size 1670, distance 11, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 12, offset 6c5b, dts 6144, size 632, distance 12, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 13, offset 6ed3, dts 6656, size 612, distance 13, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 14, offset 7137, dts 7168, size 1263, distance 14, keyframe 0 [OPENCV:FFMPEG:48] Processing st: 0, edit list 0 - media time: 0, duration: 7680 [OPENCV:FFMPEG:56] type:'udta' parent:'moov' sz: 98 713 803 [OPENCV:FFMPEG:56] type:'meta' parent:'udta' sz: 90 8 90 [OPENCV:FFMPEG:56] type:'hdlr' parent:'meta' sz: 33 8 78 [OPENCV:FFMPEG:56] ctype=[0][0][0][0] [OPENCV:FFMPEG:56] stype=mdir [OPENCV:FFMPEG:56] type:'ilst' parent:'meta' sz: 45 41 78 [OPENCV:FFMPEG:56] type:'[169]too' parent:'ilst' sz: 37 8 37 [OPENCV:FFMPEG:56] on_parse_exit_offset=31057 [OPENCV:FFMPEG:48] Before avformat_find_stream_info() pos: 31057 bytes read:31057 seeks:0 nb_streams:1 [OPENCV:FFMPEG:56] stream 0, sample 0, dts 0 [OPENCV:FFMPEG:48] Format yuv420p chosen by get_format(). [OPENCV:FFMPEG:48] All info found [OPENCV:FFMPEG:56] stream 0: start_time: 0 duration: 0.6 [OPENCV:FFMPEG:56] format: start_time: 0 duration: 0.6 (estimate from stream) bitrate=414 kb/s [OPENCV:FFMPEG:48] After avformat_find_stream_info() pos: 5112 bytes read:31057 seeks:0 frames:1 [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_impl.hpp (1022) open FFMPEG: stream[0] is video stream with codecID=167 width=322 height=242 [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (927) HWAccelIterator FFMPEG: allowed acceleration types (mfx): 'qsv.iHD' [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (945) HWAccelIterator FFMPEG: disabled codecs: 'av1.vaapi,av1_qsv,vp8.vaapi,vp8_qsv' [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_impl.hpp (1054) open FFMPEG: trying to configure H/W acceleration: 'qsv.iHD' [OPENCV:FFMPEG:40] Trying to use DRM render node for device 0. [OPENCV:FFMPEG:40] libva: VA-API version 1.9.0 libva: Trying to open /usr/lib64/dri/iHD_drv_video.so libva: Found init function __vaDriverInit_1_9 libva: va_openDriver() returns 0 Initialised VAAPI connection: version 1.9 [OPENCV:FFMPEG:48] Format 0x41524742 -> bgra. [OPENCV:FFMPEG:48] Format 0x42475241 -> argb. [OPENCV:FFMPEG:48] Format 0x41424752 -> rgba. [OPENCV:FFMPEG:48] Format 0x52474241 -> abgr. [OPENCV:FFMPEG:48] Format 0x58524742 -> bgr0. [OPENCV:FFMPEG:48] Format 0x42475258 -> 0rgb. [OPENCV:FFMPEG:48] Format 0x58424752 -> rgb0. [OPENCV:FFMPEG:48] Format 0x52474258 -> 0bgr. [OPENCV:FFMPEG:48] Format 0x30335241 -> unknown. [OPENCV:FFMPEG:48] Format 0x30334241 -> unknown. [OPENCV:FFMPEG:48] Format 0x30335258 -> unknown. [OPENCV:FFMPEG:48] Format 0x30334258 -> unknown. [OPENCV:FFMPEG:48] Format 0x36314752 -> unknown. [OPENCV:FFMPEG:48] Format 0x50424752 -> unknown. [OPENCV:FFMPEG:48] Format 0x50524742 -> unknown. [OPENCV:FFMPEG:48] Format 0x56555941 -> unknown. [OPENCV:FFMPEG:48] Format 0x30303859 -> gray. [OPENCV:FFMPEG:48] Format 0x3231564e -> nv12. [OPENCV:FFMPEG:48] Format 0x3132564e -> unknown. [OPENCV:FFMPEG:48] Format 0x32595559 -> yuyv422. [OPENCV:FFMPEG:48] Format 0x59565955 -> uyvy422. [OPENCV:FFMPEG:48] Format 0x32315659 -> yuv420p. [OPENCV:FFMPEG:48] Format 0x30323449 -> yuv420p. [OPENCV:FFMPEG:48] Format 0x50313134 -> yuv411p. [OPENCV:FFMPEG:48] Format 0x48323234 -> yuv422p. [OPENCV:FFMPEG:48] Format 0x56323234 -> yuv440p. [OPENCV:FFMPEG:48] Format 0x50343434 -> yuv444p. [OPENCV:FFMPEG:48] Format 0x33434d49 -> unknown. [OPENCV:FFMPEG:48] Format 0x30313050 -> p010le. [OPENCV:FFMPEG:40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 (). [OPENCV:FFMPEG:40] Driver not found in known nonstandard list, using standard behaviour. [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (272) hw_check_device FFMPEG: Using MFX video acceleration on device: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 () [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (562) hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for vaapi on device 'default' [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (579) hw_create_device FFMPEG: Can't bind vaapi video acceleration context to already created OpenCL context [OPENCV:FFMPEG:40] Initialize MFX session: API version is 1.34, implementation version is 1.34 [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (305) hw_create_derived_context FFMPEG: Created derived video acceleration context (av_hwdevice_ctx_create_derived) for qsv VideoCapture FFMPEG:MFX [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (618) hw_create_frames FFMPEG: avcodec_get_hw_frames_parameters() call failed: -2 [OPENCV:FFMPEG:48] Created surface 0. [OPENCV:FFMPEG:48] Direct mapping possible. [OPENCV:FFMPEG:48] Created surface 0x1. [OPENCV:FFMPEG:48] Created surface 0x2. [OPENCV:FFMPEG:48] Created surface 0x3. [OPENCV:FFMPEG:48] Created surface 0x4. [OPENCV:FFMPEG:48] Created surface 0x5. [OPENCV:FFMPEG:48] Created surface 0x6. [OPENCV:FFMPEG:48] Created surface 0x7. [OPENCV:FFMPEG:48] Created surface 0x8. [OPENCV:FFMPEG:48] Created surface 0x9. [OPENCV:FFMPEG:48] Created surface 0xa. [OPENCV:FFMPEG:48] Created surface 0xb. [OPENCV:FFMPEG:48] Created surface 0xc. [OPENCV:FFMPEG:48] Created surface 0xd. [OPENCV:FFMPEG:48] Created surface 0xe. [OPENCV:FFMPEG:48] Created surface 0xf. [OPENCV:FFMPEG:48] Created surface 0x10. [OPENCV:FFMPEG:48] Created surface 0x11. [OPENCV:FFMPEG:48] Created surface 0x12. [OPENCV:FFMPEG:48] Created surface 0x13. [OPENCV:FFMPEG:48] Created surface 0x14. [OPENCV:FFMPEG:48] Created surface 0x15. [OPENCV:FFMPEG:48] Created surface 0x16. [OPENCV:FFMPEG:48] Created surface 0x17. [OPENCV:FFMPEG:48] Created surface 0x18. [OPENCV:FFMPEG:48] Created surface 0x19. [OPENCV:FFMPEG:48] Created surface 0x1a. [OPENCV:FFMPEG:48] Created surface 0x1b. [OPENCV:FFMPEG:48] Created surface 0x1c. [OPENCV:FFMPEG:48] Created surface 0x1d. [OPENCV:FFMPEG:48] Created surface 0x1e. [OPENCV:FFMPEG:48] Created surface 0x1f. [OPENCV:FFMPEG:48] Format qsv chosen by get_format(). [OPENCV:FFMPEG:40] Decoder: output is video memory surface [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (618) hw_create_frames FFMPEG: avcodec_get_hw_frames_parameters() call failed: -2 [OPENCV:FFMPEG:48] Created surface 0x20. [OPENCV:FFMPEG:48] Direct mapping possible. [OPENCV:FFMPEG:48] Created surface 0x21. [OPENCV:FFMPEG:48] Created surface 0x22. [OPENCV:FFMPEG:48] Created surface 0x23. [OPENCV:FFMPEG:48] Created surface 0x24. [OPENCV:FFMPEG:48] Created surface 0x25. [OPENCV:FFMPEG:48] Created surface 0x26. [OPENCV:FFMPEG:48] Created surface 0x27. [OPENCV:FFMPEG:48] Created surface 0x28. [OPENCV:FFMPEG:48] Created surface 0x29. [OPENCV:FFMPEG:48] Created surface 0x2a. [OPENCV:FFMPEG:48] Created surface 0x2b. [OPENCV:FFMPEG:48] Created surface 0x2c. [OPENCV:FFMPEG:48] Created surface 0x2d. [OPENCV:FFMPEG:48] Created surface 0x2e. [OPENCV:FFMPEG:48] Created surface 0x2f. [OPENCV:FFMPEG:48] Created surface 0x30. [OPENCV:FFMPEG:48] Created surface 0x31. [OPENCV:FFMPEG:48] Created surface 0x32. [OPENCV:FFMPEG:48] Created surface 0x33. [OPENCV:FFMPEG:48] Created surface 0x34. [OPENCV:FFMPEG:48] Created surface 0x35. [OPENCV:FFMPEG:48] Created surface 0x36. [OPENCV:FFMPEG:48] Created surface 0x37. [OPENCV:FFMPEG:48] Created surface 0x38. [OPENCV:FFMPEG:48] Created surface 0x39. [OPENCV:FFMPEG:48] Created surface 0x3a. [OPENCV:FFMPEG:48] Created surface 0x3b. [OPENCV:FFMPEG:48] Created surface 0x3c. [OPENCV:FFMPEG:48] Created surface 0x3d. [OPENCV:FFMPEG:48] Created surface 0x3e. [OPENCV:FFMPEG:48] Created surface 0x3f. [OPENCV:FFMPEG:48] Format qsv chosen by get_format(). [OPENCV:FFMPEG:40] Decoder: output is video memory surface [OPENCV:FFMPEG:16] Error initializing the MFX video decoder: unsupported (-3) /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:743: Failure Value of: hw_reader.read(frame) Actual: false Expected: true Google Test trace: /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:732: frame=0 /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:745: Failure Value of: frame.empty() Actual: true Expected: false Google Test trace: /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:732: frame=0 [OPENCV:FFMPEG:40] Statistics: 31057 bytes read, 0 seeks [ FAILED ] videoio/videocapture_acceleration.read/84, where GetParam() = (sample_322x242_15frames.yuv420p.libvpx-vp9.mp4, FFMPEG, MFX, false) (6 ms)

Master branch

[ RUN ] videoio/videocapture_acceleration.read/84, where GetParam() = (sample_322x242_15frames.yuv420p.libvpx-vp9.mp4, FFMPEG, MFX, false) [OPENCV:FFMPEG:48] Opening '/home/alalek/projects/opencv/extra/testdata/highgui/video/sample_322x242_15frames.yuv420p.libvpx-vp9.mp4' for reading [OPENCV:FFMPEG:48] Setting default whitelist 'file,crypto,data' [OPENCV:FFMPEG:56] Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:2048 [OPENCV:FFMPEG:56] Probing mp3 score:1 size:2048 [OPENCV:FFMPEG:48] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100 [OPENCV:FFMPEG:56] type:'ftyp' parent:'root' sz: 28 8 31057 [OPENCV:FFMPEG:48] ISO: File Type Major Brand: isom [OPENCV:FFMPEG:56] type:'free' parent:'root' sz: 8 36 31057 [OPENCV:FFMPEG:56] type:'mdat' parent:'root' sz: 30210 44 31057 [OPENCV:FFMPEG:56] type:'moov' parent:'root' sz: 811 30254 31057 [OPENCV:FFMPEG:56] type:'mvhd' parent:'moov' sz: 108 8 803 [OPENCV:FFMPEG:56] time scale = 1000 [OPENCV:FFMPEG:56] type:'trak' parent:'moov' sz: 597 116 803 [OPENCV:FFMPEG:56] type:'tkhd' parent:'trak' sz: 92 8 589 [OPENCV:FFMPEG:56] type:'edts' parent:'trak' sz: 36 100 589 [OPENCV:FFMPEG:56] type:'elst' parent:'edts' sz: 28 8 28 [OPENCV:FFMPEG:56] track[0].edit_count = 1 [OPENCV:FFMPEG:56] duration=600 time=0 rate=1.000000 [OPENCV:FFMPEG:56] type:'mdia' parent:'trak' sz: 461 136 589 [OPENCV:FFMPEG:56] type:'mdhd' parent:'mdia' sz: 32 8 453 [OPENCV:FFMPEG:56] type:'hdlr' parent:'mdia' sz: 45 40 453 [OPENCV:FFMPEG:56] ctype=[0][0][0][0] [OPENCV:FFMPEG:56] stype=vide [OPENCV:FFMPEG:56] type:'minf' parent:'mdia' sz: 376 85 453 [OPENCV:FFMPEG:56] type:'vmhd' parent:'minf' sz: 20 8 368 [OPENCV:FFMPEG:56] type:'dinf' parent:'minf' sz: 36 28 368 [OPENCV:FFMPEG:56] type:'dref' parent:'dinf' sz: 28 8 28 [OPENCV:FFMPEG:48] Unknown dref type 0x206c7275 size 12 [OPENCV:FFMPEG:56] type:'stbl' parent:'minf' sz: 312 64 368 [OPENCV:FFMPEG:56] type:'stsd' parent:'stbl' sz: 132 8 304 [OPENCV:FFMPEG:56] size=116 4CC=vp09 codec_type=0 [OPENCV:FFMPEG:56] type:'vpcC' parent:'stsd' sz: 20 8 30 [OPENCV:FFMPEG:56] type:'fiel' parent:'stsd' sz: 10 28 30 [OPENCV:FFMPEG:56] type:'stts' parent:'stbl' sz: 24 140 304 [OPENCV:FFMPEG:56] track[0].stts.entries = 1 [OPENCV:FFMPEG:56] sample_count=15, sample_duration=512 [OPENCV:FFMPEG:56] type:'stss' parent:'stbl' sz: 20 164 304 [OPENCV:FFMPEG:56] keyframe_count = 1 [OPENCV:FFMPEG:56] type:'stsc' parent:'stbl' sz: 28 184 304 [OPENCV:FFMPEG:56] track[0].stsc.entries = 1 [OPENCV:FFMPEG:56] type:'stsz' parent:'stbl' sz: 80 212 304 [OPENCV:FFMPEG:56] sample_size = 0 sample_count = 15 [OPENCV:FFMPEG:56] type:'stco' parent:'stbl' sz: 20 292 304 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 0, offset 2c, dts 0, size 5068, distance 0, keyframe 1 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 1, offset 13f8, dts 512, size 1308, distance 1, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 2, offset 1914, dts 1024, size 1265, distance 2, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 3, offset 1e05, dts 1536, size 1507, distance 3, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 4, offset 23e8, dts 2048, size 2089, distance 4, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 5, offset 2c11, dts 2560, size 1956, distance 5, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 6, offset 33b5, dts 3072, size 1795, distance 6, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 7, offset 3ab8, dts 3584, size 2204, distance 7, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 8, offset 4354, dts 4096, size 2267, distance 8, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 9, offset 4c2f, dts 4608, size 1315, distance 9, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 10, offset 5152, dts 5120, size 5251, distance 10, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 11, offset 65d5, dts 5632, size 1670, distance 11, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 12, offset 6c5b, dts 6144, size 632, distance 12, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 13, offset 6ed3, dts 6656, size 612, distance 13, keyframe 0 [OPENCV:FFMPEG:56] AVIndex stream 0, sample 14, offset 7137, dts 7168, size 1263, distance 14, keyframe 0 [OPENCV:FFMPEG:48] Processing st: 0, edit list 0 - media time: 0, duration: 7680 [OPENCV:FFMPEG:56] type:'udta' parent:'moov' sz: 98 713 803 [OPENCV:FFMPEG:56] type:'meta' parent:'udta' sz: 90 8 90 [OPENCV:FFMPEG:56] type:'hdlr' parent:'meta' sz: 33 8 78 [OPENCV:FFMPEG:56] ctype=[0][0][0][0] [OPENCV:FFMPEG:56] stype=mdir [OPENCV:FFMPEG:56] type:'ilst' parent:'meta' sz: 45 41 78 [OPENCV:FFMPEG:56] type:'[169]too' parent:'ilst' sz: 37 8 37 [OPENCV:FFMPEG:56] on_parse_exit_offset=31057 [OPENCV:FFMPEG:48] Before avformat_find_stream_info() pos: 31057 bytes read:31057 seeks:0 nb_streams:1 [OPENCV:FFMPEG:56] stream 0, sample 0, dts 0 [OPENCV:FFMPEG:48] Format yuv420p chosen by get_format(). [OPENCV:FFMPEG:48] All info found [OPENCV:FFMPEG:56] stream 0: start_time: 0 duration: 0.6 [OPENCV:FFMPEG:56] format: start_time: 0 duration: 0.6 (estimate from stream) bitrate=414 kb/s [OPENCV:FFMPEG:48] After avformat_find_stream_info() pos: 5112 bytes read:31057 seeks:0 frames:1 [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_impl.hpp (1022) open FFMPEG: stream[0] is video stream with codecID=167 width=322 height=242 [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (927) HWAccelIterator FFMPEG: allowed acceleration types (mfx): 'qsv.iHD' [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (945) HWAccelIterator FFMPEG: disabled codecs: 'av1.vaapi,av1_qsv,vp8.vaapi,vp8_qsv' [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_impl.hpp (1054) open FFMPEG: trying to configure H/W acceleration: 'qsv.iHD' [OPENCV:FFMPEG:40] Trying to use DRM render node for device 0. [OPENCV:FFMPEG:40] libva: VA-API version 1.9.0 libva: Trying to open /usr/lib64/dri/iHD_drv_video.so libva: Found init function __vaDriverInit_1_9 libva: va_openDriver() returns 0 Initialised VAAPI connection: version 1.9 [OPENCV:FFMPEG:48] Format 0x41524742 -> bgra. [OPENCV:FFMPEG:48] Format 0x42475241 -> argb. [OPENCV:FFMPEG:48] Format 0x41424752 -> rgba. [OPENCV:FFMPEG:48] Format 0x52474241 -> abgr. [OPENCV:FFMPEG:48] Format 0x58524742 -> bgr0. [OPENCV:FFMPEG:48] Format 0x42475258 -> 0rgb. [OPENCV:FFMPEG:48] Format 0x58424752 -> rgb0. [OPENCV:FFMPEG:48] Format 0x52474258 -> 0bgr. [OPENCV:FFMPEG:48] Format 0x30335241 -> unknown. [OPENCV:FFMPEG:48] Format 0x30334241 -> unknown. [OPENCV:FFMPEG:48] Format 0x30335258 -> unknown. [OPENCV:FFMPEG:48] Format 0x30334258 -> unknown. [OPENCV:FFMPEG:48] Format 0x36314752 -> unknown. [OPENCV:FFMPEG:48] Format 0x50424752 -> unknown. [OPENCV:FFMPEG:48] Format 0x50524742 -> unknown. [OPENCV:FFMPEG:48] Format 0x56555941 -> unknown. [OPENCV:FFMPEG:48] Format 0x30303859 -> gray. [OPENCV:FFMPEG:48] Format 0x3231564e -> nv12. [OPENCV:FFMPEG:48] Format 0x3132564e -> unknown. [OPENCV:FFMPEG:48] Format 0x32595559 -> yuyv422. [OPENCV:FFMPEG:48] Format 0x59565955 -> uyvy422. [OPENCV:FFMPEG:48] Format 0x32315659 -> yuv420p. [OPENCV:FFMPEG:48] Format 0x30323449 -> yuv420p. [OPENCV:FFMPEG:48] Format 0x50313134 -> yuv411p. [OPENCV:FFMPEG:48] Format 0x48323234 -> yuv422p. [OPENCV:FFMPEG:48] Format 0x56323234 -> yuv440p. [OPENCV:FFMPEG:48] Format 0x50343434 -> yuv444p. [OPENCV:FFMPEG:48] Format 0x33434d49 -> unknown. [OPENCV:FFMPEG:48] Format 0x30313050 -> p010le. [OPENCV:FFMPEG:40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 (). [OPENCV:FFMPEG:40] Driver not found in known nonstandard list, using standard behaviour. [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (272) hw_check_device FFMPEG: Using MFX video acceleration on device: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 () [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (562) hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for vaapi on device 'default' [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (579) hw_create_device FFMPEG: Can't bind vaapi video acceleration context to already created OpenCL context [OPENCV:FFMPEG:40] Initialize MFX session: API version is 1.34, implementation version is 1.34 [ INFO:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (305) hw_create_derived_context FFMPEG: Created derived video acceleration context (av_hwdevice_ctx_create_derived) for qsv VideoCapture FFMPEG:MFX [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (618) hw_create_frames FFMPEG: avcodec_get_hw_frames_parameters() call failed: -2 [OPENCV:FFMPEG:48] Created surface 0. [OPENCV:FFMPEG:48] Direct mapping possible. [OPENCV:FFMPEG:48] Created surface 0x1. [OPENCV:FFMPEG:48] Created surface 0x2. [OPENCV:FFMPEG:48] Created surface 0x3. [OPENCV:FFMPEG:48] Created surface 0x4. [OPENCV:FFMPEG:48] Created surface 0x5. [OPENCV:FFMPEG:48] Created surface 0x6. [OPENCV:FFMPEG:48] Created surface 0x7. [OPENCV:FFMPEG:48] Created surface 0x8. [OPENCV:FFMPEG:48] Created surface 0x9. [OPENCV:FFMPEG:48] Created surface 0xa. [OPENCV:FFMPEG:48] Created surface 0xb. [OPENCV:FFMPEG:48] Created surface 0xc. [OPENCV:FFMPEG:48] Created surface 0xd. [OPENCV:FFMPEG:48] Created surface 0xe. [OPENCV:FFMPEG:48] Created surface 0xf. [OPENCV:FFMPEG:48] Created surface 0x10. [OPENCV:FFMPEG:48] Created surface 0x11. [OPENCV:FFMPEG:48] Created surface 0x12. [OPENCV:FFMPEG:48] Created surface 0x13. [OPENCV:FFMPEG:48] Created surface 0x14. [OPENCV:FFMPEG:48] Created surface 0x15. [OPENCV:FFMPEG:48] Created surface 0x16. [OPENCV:FFMPEG:48] Created surface 0x17. [OPENCV:FFMPEG:48] Created surface 0x18. [OPENCV:FFMPEG:48] Created surface 0x19. [OPENCV:FFMPEG:48] Created surface 0x1a. [OPENCV:FFMPEG:48] Created surface 0x1b. [OPENCV:FFMPEG:48] Created surface 0x1c. [OPENCV:FFMPEG:48] Created surface 0x1d. [OPENCV:FFMPEG:48] Created surface 0x1e. [OPENCV:FFMPEG:48] Created surface 0x1f. [OPENCV:FFMPEG:48] Format qsv chosen by get_format(). [OPENCV:FFMPEG:40] Decoder: output is video memory surface [DEBUG:0] global /home/alalek/projects/opencv/dev/modules/videoio/src/cap_ffmpeg_hw.hpp (618) hw_create_frames FFMPEG: avcodec_get_hw_frames_parameters() call failed: -2 [OPENCV:FFMPEG:48] Created surface 0x20. [OPENCV:FFMPEG:48] Direct mapping possible. [OPENCV:FFMPEG:48] Created surface 0x21. [OPENCV:FFMPEG:48] Created surface 0x22. [OPENCV:FFMPEG:48] Created surface 0x23. [OPENCV:FFMPEG:48] Created surface 0x24. [OPENCV:FFMPEG:48] Created surface 0x25. [OPENCV:FFMPEG:48] Created surface 0x26. [OPENCV:FFMPEG:48] Created surface 0x27. [OPENCV:FFMPEG:48] Created surface 0x28. [OPENCV:FFMPEG:48] Created surface 0x29. [OPENCV:FFMPEG:48] Created surface 0x2a. [OPENCV:FFMPEG:48] Created surface 0x2b. [OPENCV:FFMPEG:48] Created surface 0x2c. [OPENCV:FFMPEG:48] Created surface 0x2d. [OPENCV:FFMPEG:48] Created surface 0x2e. [OPENCV:FFMPEG:48] Created surface 0x2f. [OPENCV:FFMPEG:48] Created surface 0x30. [OPENCV:FFMPEG:48] Created surface 0x31. [OPENCV:FFMPEG:48] Created surface 0x32. [OPENCV:FFMPEG:48] Created surface 0x33. [OPENCV:FFMPEG:48] Created surface 0x34. [OPENCV:FFMPEG:48] Created surface 0x35. [OPENCV:FFMPEG:48] Created surface 0x36. [OPENCV:FFMPEG:48] Created surface 0x37. [OPENCV:FFMPEG:48] Created surface 0x38. [OPENCV:FFMPEG:48] Created surface 0x39. [OPENCV:FFMPEG:48] Created surface 0x3a. [OPENCV:FFMPEG:48] Created surface 0x3b. [OPENCV:FFMPEG:48] Created surface 0x3c. [OPENCV:FFMPEG:48] Created surface 0x3d. [OPENCV:FFMPEG:48] Created surface 0x3e. [OPENCV:FFMPEG:48] Created surface 0x3f. [OPENCV:FFMPEG:48] Format qsv chosen by get_format(). [OPENCV:FFMPEG:40] Decoder: output is video memory surface [OPENCV:FFMPEG:16] Error initializing the MFX video decoder: unsupported (-3) /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:743: Failure Value of: hw_reader.read(frame) Actual: false Expected: true Google Test trace: /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:732: frame=0 /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:745: Failure Value of: frame.empty() Actual: true Expected: false Google Test trace: /home/alalek/projects/opencv/dev/modules/videoio/test/test_video_io.cpp:732: frame=0 [OPENCV:FFMPEG:40] Statistics: 31057 bytes read, 0 seeks [ FAILED ] videoio/videocapture_acceleration.read/84, where GetParam() = (sample_322x242_15frames.yuv420p.libvpx-vp9.mp4, FFMPEG, MFX, false) (6 ms)

vainfo

$ vainfo libva info: VA-API version 1.9.0 libva info: Trying to open /usr/lib64/dri/iHD_drv_video.so libva info: Found init function __vaDriverInit_1_9 libva info: va_openDriver() returns 0 vainfo: VA-API version: 1.9 (libva 2.9.0) vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.3.0 () vainfo: Supported profile and entrypoints VAProfileNone : VAEntrypointVideoProc VAProfileNone : VAEntrypointStats VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Simple : VAEntrypointEncSlice VAProfileMPEG2Main : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointEncSlice VAProfileH264Main : VAEntrypointVLD VAProfileH264Main : VAEntrypointEncSlice VAProfileH264Main : VAEntrypointFEI VAProfileH264Main : VAEntrypointEncSliceLP VAProfileH264High : VAEntrypointVLD VAProfileH264High : VAEntrypointEncSlice VAProfileH264High : VAEntrypointFEI VAProfileH264High : VAEntrypointEncSliceLP VAProfileVC1Simple : VAEntrypointVLD VAProfileVC1Main : VAEntrypointVLD VAProfileVC1Advanced : VAEntrypointVLD VAProfileJPEGBaseline : VAEntrypointVLD VAProfileJPEGBaseline : VAEntrypointEncPicture VAProfileH264ConstrainedBaseline: VAEntrypointVLD VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice VAProfileH264ConstrainedBaseline: VAEntrypointFEI VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP VAProfileVP8Version0_3 : VAEntrypointVLD VAProfileHEVCMain : VAEntrypointVLD VAProfileHEVCMain : VAEntrypointEncSlice VAProfileHEVCMain : VAEntrypointFEI

Looks like there is no VP9 entry in vainfo:

Fedora 33

intel-media-driver.x86_64 20.3.0-2.fc33 @rpmfusion-nonfree-updates

CPU: i5-6600 (Skylake)

Similar problem: https://bugs.launchpad.net/ubuntu/+source/intel-vaapi-driver/+bug/1591714

@mikhail-nikolskiy Can we handle/workaround this somehow?

No easy workaround. ffmpeg-qsv decoder designed with delayed initialization, the MFXVideoDECODE_Init called on decoding first frame, it's too late for fallback to SW decoder in current implementation. No such problem in VAAPI and D3D11 decoders as they report all errors on opening stage (av_codec_open2 call).
Disable MFX decode (revert "qsv.iHD" back to empty string "") or keep as known issue in case MFX acceleration explicitly requested?

@mikhail-nikolskiy Thank you for the information!

Looks like we need to add the first frame grabbing as part of .open() implementation to handle/workaround this case (I believe we can implement this later).

alalek · 2021-05-05T02:48:32Z

@mshabunin Could you please take a look on CMake scripts changes?

Note: "plugins" build mode is not fully supported yet (but we should ensure that compilation is not broken).

CMakeLists.txt

alalek

Great contribution! Thank you 👍

modules/videoio/cmake/detect_ffmpeg.cmake

cv::UMat output/input in VideoCapture/VideoWriter (data stays in GPU memory) * FFMPEG with UMat input/output * OpenCL_D3D* context * fix Linux build * cosmetic changes * fix build if USE_AV_HW_CODECS=0 * simplify how child context pointer stored in parent context * QSV interop with OpenCL on Windows * detect_msdk.cmake via pkg-config * fix av_buffer_ref() usage * revert windows-decode-mfx whitelisting; remove debug msg * address review comments * rename property to HW_ACCELERATION_USE_OPENCL * fix issue with "cl_khr_d3d11_sharing" extension not reported by OpenCL GPU+CPU platform * core(ocl): add OpenCL stubs for configurations without OpenCL * videoio(ffmpeg): update #if guards * Put OpenCL related code under HAVE_OPENCL; simplify reuse of media context from OpenCL context * videoio(test): skip unsupported tests - plugins don't support OpenCL/UMat yet - change handling of *_USE_OPENCL flag * videoio(ffmpeg): OpenCL dependency * videoio(ffmpeg): MediaSDK/oneVPL dependency * cleanup, logging * cmake: fix handling of 3rdparty interface targets Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>

alalek mentioned this pull request Mar 20, 2021

OE-XX: Hardware-accelerated Video I/O #19676

Open

This was referenced Mar 24, 2021

Performance optimization in DirectX and VAAPI interop #19776

Closed

Performance optimization in DirectX and VAAPI interop #19783

Merged

mikhail-nikolskiy commented Mar 26, 2021

View reviewed changes

alalek reviewed Apr 8, 2021

View reviewed changes

mikhail-nikolskiy and others added 10 commits April 8, 2021 13:40

FFMPEG with UMat input/output

f45a15c

OpenCL_D3D* context

0c03656

fix Linux build

4065a4b

cosmetic changes

db2966b

fix build if USE_AV_HW_CODECS=0

b3af523

simplify how child context pointer stored in parent context

a094be4

QSV interop with OpenCL on Windows

6c821d1

detect_msdk.cmake via pkg-config

54758cf

fix av_buffer_ref() usage

221c605

revert windows-decode-mfx whitelisting; remove debug msg

cb3d9e0

mikhail-nikolskiy force-pushed the ffmpeg-umat branch from 4b015ec to 1d9e6e6 Compare April 10, 2021 00:22

mikhail-nikolskiy added 2 commits April 12, 2021 19:50

address review comments

fadaa09

rename property to HW_ACCELERATION_USE_OPENCL

c6e2839

fix issue with "cl_khr_d3d11_sharing" extension not reported by OpenC…

f28c404

…L GPU+CPU platform

alalek and others added 3 commits April 13, 2021 17:50

core(ocl): add OpenCL stubs for configurations without OpenCL

d38618e

videoio(ffmpeg): update #if guards

0fa52cf

Put OpenCL related code under HAVE_OPENCL; simplify reuse of media co…

44d82a9

…ntext from OpenCL context

alalek reviewed Apr 14, 2021

View reviewed changes

alalek mentioned this pull request Apr 19, 2021

videoio(plugin): add query API for plugins #19934

Merged

Merge branch 'upstream/master'

142b68c

alalek reviewed Apr 26, 2021

View reviewed changes

modules/core/src/directx.cpp Show resolved Hide resolved

alalek mentioned this pull request Apr 30, 2021

core(tls): handle process termination / cleanup issues #20014

Merged

alalek added 3 commits May 1, 2021 23:53

videoio(test): skip unsupported tests

237710f

- plugins don't support OpenCL/UMat yet - change hanling of *_USE_OPENCL flag

Merge branch 'upstream/master'

471761a

videoio(ffmpeg): OpenCL dependency

bc905b4

alalek reviewed May 3, 2021

View reviewed changes

mshabunin reviewed May 7, 2021

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

alalek added 2 commits May 8, 2021 16:29

videoio(ffmpeg): MediaSDK/oneVPL dependency

f62517e

cleanup, logging

0dec367

alalek approved these changes May 12, 2021

View reviewed changes

mshabunin reviewed May 12, 2021

View reviewed changes

modules/videoio/cmake/detect_ffmpeg.cmake Show resolved Hide resolved

cmake: fix handling of 3rdparty interface targets

0465d83

mshabunin approved these changes May 14, 2021

View reviewed changes

alalek merged commit a604d44 into opencv:master May 14, 2021

alalek mentioned this pull request Jun 4, 2021

(5.x) Merge 4.x #20216

Merged

kallaballa mentioned this pull request Nov 17, 2022

Redesign of CL/VA interop initialization #22780

Open

6 tasks

Uh oh!

Conversation

mikhail-nikolskiy commented Mar 20, 2021 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

mikhail-nikolskiy commented Mar 26, 2021

Uh oh!

mikhail-nikolskiy Mar 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikhail-nikolskiy commented Mar 26, 2021

Uh oh!

asmorkalov commented Mar 29, 2021

Uh oh!

asmorkalov commented Mar 29, 2021

Uh oh!

asenyaev commented Apr 5, 2021

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asenyaev commented Apr 8, 2021

Uh oh!

mikhail-nikolskiy commented Apr 9, 2021

Uh oh!

alalek commented Apr 9, 2021

Uh oh!

alalek commented Apr 9, 2021

Uh oh!

mikhail-nikolskiy commented Apr 10, 2021

Uh oh!

alalek commented Apr 12, 2021

Uh oh!

mikhail-nikolskiy commented Apr 13, 2021

Uh oh!

mikhail-nikolskiy commented Apr 14, 2021

Uh oh!

alalek commented Apr 14, 2021

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dmatveev commented Apr 28, 2021

Uh oh!

mikhail-nikolskiy commented Apr 29, 2021 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mikhail-nikolskiy commented Mar 20, 2021 •

edited by alalek

Loading

mikhail-nikolskiy Mar 26, 2021 •

edited

Loading

mikhail-nikolskiy commented Apr 29, 2021 •

edited by alalek

Loading

alalek commented May 5, 2021 •

edited

Loading