VideoCapture with FFmpeg and CUDA Hardware Acceleration Returns Empty Frames in OpenCV 4.7.0 #25185
Description
System Information
Operating System: CentOS Linux release 7.9.2009
Compiler: gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
OpenCV Version: 4.7.0
FFmpeg Version: 6.0.1
GPU: NVIDIA A10, Driver Version: 470.161.03, CUDA Version: 11.4
Detailed description
After compiling FFmpeg with hardware acceleration support and OpenCV, using cv::VideoCapture to decode a video does not work as expected. The decoded video frames are consistently empty when hardware acceleration is enabled. Using FFmpeg command-line tools, hardware-accelerated decoding functions correctly, indicating that the issue is specific to OpenCV's use of FFmpeg with hardware acceleration.
The issue does not occur when OPENCV_FFMPEG_CAPTURE_OPTIONS is not set to use CUDA hardware decoders (hw_decoders_any;cuda), but this setup does not utilize GPU acceleration, which is needed.
Steps to reproduce
1.Compile FFmpeg with CUDA hardware acceleration support.
2.Compile OpenCV 4.7.0 with FFmpeg and CUDA enabled.
3.Run the following C++ code to attempt to decode a video using cv::VideoCapture with CUDA hardware acceleration enabled:
int main(int argc, char** argv) {
int cnt = 100;
if (argc >= 2) {
std::istringstream iss(argv[1]);
iss >> cnt;
}
cv::VideoCapture cap;
bool isOpened = cap.open("../resource/background.mp4", cv::CAP_FFMPEG, {cv::CAP_PROP_HW_ACCELERATION, cv::VIDEO_ACCELERATION_ANY});
if (!isOpened) {
LOG(ERROR) << "Error opening video file";
return -1;
}
int i = 0;
cv::Mat frame;
auto next = std::chrono::steady_clock::now();
while (i++ <= cnt) {
cap >> frame;
if (frame.empty()) {
LOG(ERROR) << "Frame is empty";
cap.release();
isOpened = cap.open("../resource/background.mp4", cv::CAP_FFMPEG, {cv::CAP_PROP_HW_ACCELERATION, cv::VIDEO_ACCELERATION_ANY});
if (!isOpened) {
LOG(ERROR) << "Error reopening video file";
return -1;
}
}
LOG(INFO) << "Decode video frame cnt: " << i;
next += std::chrono::milliseconds(30);
std::this_thread::sleep_until(next);
}
return 0;
}
4.Observe that the decoded frames are empty, as indicated by the logs and error messages.
[DEBUG:0@0.002] global videoio_registry.cpp:206 VideoBackendRegistry VIDEOIO: Builtin backends(8): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); UEYE(940); OBSENSOR(930)
[DEBUG:0@0.002] global videoio_registry.cpp:230 VideoBackendRegistry VIDEOIO: Available backends(8): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); UEYE(940); OBSENSOR(930)
[ INFO:0@0.002] global videoio_registry.cpp:232 VideoBackendRegistry VIDEOIO: Enabled backends(8, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); UEYE(940); OBSENSOR(930)
[DEBUG:0@0.002] global cap_ffmpeg_impl.hpp:1111 open VIDEOIO/FFMPEG: using capture options from environment: hw_decoders_any;cuda
[OPENCV:FFMPEG:40] Reinit context to 1920x1088, pix_fmt: yuv420p
[DEBUG:0@0.026] global cap_ffmpeg_impl.hpp:1168 open FFMPEG: stream[1] is video stream with codecID=27 width=1920 height=1080
[DEBUG:0@0.026] global cap_ffmpeg_hw.hpp:934 HWAccelIterator FFMPEG: allowed acceleration types (any): 'cuda,'
[DEBUG:0@0.026] global cap_ffmpeg_hw.hpp:952 HWAccelIterator FFMPEG: disabled codecs: 'av1.vaapi,av1_qsv,vp8.vaapi,vp8_qsv'
[DEBUG:0@0.026] global cap_ffmpeg_impl.hpp:1190 open FFMPEG: trying to configure H/W acceleration: 'cuda'
[ INFO:0@0.078] global cap_ffmpeg_hw.hpp:279 hw_check_device FFMPEG: Using cuda video acceleration
[ INFO:0@0.078] global cap_ffmpeg_hw.hpp:567 hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for cuda on device 'default'
[ INFO:0@0.078] global ocl.cpp:1186 haveOpenCL Initialize OpenCL runtime...
[ INFO:0@0.079] global ocl.cpp:1192 haveOpenCL OpenCL: found 0 platforms
[OPENCV:FFMPEG:40] NVDEC capabilities:
[OPENCV:FFMPEG:40] format supported: yes, max_mb_count: 65536
[OPENCV:FFMPEG:40] min_width: 48, max_width: 4096
[OPENCV:FFMPEG:40] min_height: 16, max_height: 4096
[OPENCV:FFMPEG:40] Reinit context to 1920x1088, pix_fmt: cuda
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0308 08:03:03.889551 25258 video_decode_test.cpp:31] Frame is empty
[OPENCV:FFMPEG:40] Statistics: 1153055 bytes read, 31 seeks
[DEBUG:0@0.160] global cap_ffmpeg_impl.hpp:1111 open VIDEOIO/FFMPEG: using capture options from environment: hw_decoders_any;cuda
[OPENCV:FFMPEG:40] Reinit context to 1920x1088, pix_fmt: yuv420p
[DEBUG:0@0.179] global cap_ffmpeg_impl.hpp:1168 open FFMPEG: stream[1] is video stream with codecID=27 width=1920 height=1080
[DEBUG:0@0.179] global cap_ffmpeg_hw.hpp:934 HWAccelIterator FFMPEG: allowed acceleration types (any): 'cuda,'
[DEBUG:0@0.179] global cap_ffmpeg_hw.hpp:952 HWAccelIterator FFMPEG: disabled codecs: 'av1.vaapi,av1_qsv,vp8.vaapi,vp8_qsv'
[DEBUG:0@0.179] global cap_ffmpeg_impl.hpp:1190 open FFMPEG: trying to configure H/W acceleration: 'cuda'
[ INFO:0@0.222] global cap_ffmpeg_hw.hpp:279 hw_check_device FFMPEG: Using cuda video acceleration
[ INFO:0@0.222] global cap_ffmpeg_hw.hpp:567 hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for cuda on device 'default'
I0308 08:03:03.990667 25258 video_decode_test.cpp:43] Decode video frame cnt: 1
[OPENCV:FFMPEG:40] NVDEC capabilities:
[OPENCV:FFMPEG:40] format supported: yes, max_mb_count: 65536
[OPENCV:FFMPEG:40] min_width: 48, max_width: 4096
[OPENCV:FFMPEG:40] min_height: 16, max_height: 4096
[OPENCV:FFMPEG:40] Reinit context to 1920x1088, pix_fmt: cuda
E0308 08:03:04.028532 25258 video_decode_test.cpp:31] Frame is empty
[OPENCV:FFMPEG:40] Statistics: 1153055 bytes read, 31 seeks
Issue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- I updated to the latest OpenCV version and the issue is still there
- There is reproducer code and related data files (videos, images, onnx, etc)