videoio: Incorrect pts in encapsulated video which uses B-frames #25873
Description
System Information
OpenCV version: 4.10.0
OS: Windows 11
Compiler: Visual Studio 2022
Detailed description
Raw encoded video streams encapsulated with cv::VideoWriter don't have the correct presentation timestamps when bidirectional B-Frames are used. e.g. A GOP of IBP which is decoded as IPB (because the B-frames need information from the P-frames) currently doesn't have the correct presentation timestamps set because they are set to the decoding not presentation order inside cv::VideoWriter.
This produced files are readable and pass the current tests using cv::VideoReader and play in some media players (e.g. vlc) but not in others (e.g. quicktime).
Example
The output from ffprobe on the file produced by encapsulating the raw video stream taken from video/sample_322x242_15frames.yuv420p.libx264.mp4 in the videoio_encapsulate.write test shows a presentation order (pts) of IPB
"frames": [
{ "key_frame": 1, "pts": 0, "pts_time": "0.000000", "pkt_dts": 1024, "pkt_pos": "48", "pkt_size": "6615", "pict_type": "I",
"side_data_list": [
{ }
] },
{ "key_frame": 0, "pts": 1024, "pts_time": "0.080000", "pkt_dts": 1536, "pkt_pos": "10747", "pkt_size": "1193", "pict_type": "B" },
{ "key_frame": 0, "pts": 512, "pts_time": "0.040000", "pkt_dts": 2048, "pkt_pos": "6663", "pkt_size": "4084", "pict_type": "P" },
{ "key_frame": 0, "pts": 2048, "pts_time": "0.160000", "pkt_dts": 2560, "pkt_pos": "14848", "pkt_size": "2247", "pict_type": "B" },
{ "key_frame": 0, "pts": 2560, "pts_time": "0.200000", "pkt_dts": 3072, "pkt_pos": "17095", "pkt_size": "1971", "pict_type": "B" },
{ "key_frame": 0, "pts": 1536, "pts_time": "0.120000", "pkt_dts": 3584, "pkt_pos": "11940", "pkt_size": "2908", "pict_type": "P" },
{ "key_frame": 0, "pts": 3584, "pts_time": "0.280000", "pkt_dts": 4096, "pkt_pos": "21651", "pkt_size": "2445", "pict_type": "B" },
{ "key_frame": 0, "pts": 4096, "pts_time": "0.320000", "pkt_dts": 4608, "pkt_pos": "24096", "pkt_size": "2097", "pict_type": "B" },
{ "key_frame": 0, "pts": 3072, "pts_time": "0.240000", "pkt_dts": 5120, "pkt_pos": "19066", "pkt_size": "2585", "pict_type": "P" },
{ "key_frame": 0, "pts": 5120, "pts_time": "0.400000", "pkt_dts": 5632, "pkt_pos": "30169", "pkt_size": "2430", "pict_type": "B" },
{ "key_frame": 0, "pts": 5632, "pts_time": "0.440000", "pkt_dts": 6144, "pkt_pos": "32599", "pkt_size": "1438", "pict_type": "B" },
{ "key_frame": 0, "pts": 4608, "pts_time": "0.360000", "pkt_dts": 6656, "pkt_pos": "26193", "pkt_size": "3976", "pict_type": "P" },
{ "key_frame": 0, "pts": 6656, "pts_time": "0.520000", "pkt_dts": 7168, "pkt_pos": "36043", "pkt_size": "2339", "pict_type": "B" },
{ "key_frame": 0, "pts": 7168, "pts_time": "0.560000", "pkt_pos": "38382", "pkt_size": "1442", "pict_type": "B" },
{ "key_frame": 0, "pts": 6144, "pts_time": "0.480000", "pkt_pos": "34037", "pkt_size": "2006", "pict_type": "P" }
with the correct presentation order being IBP is shown below
"frames": [
{ "key_frame": 1, "pts": 0, "pts_time": "0.000000", "pkt_dts": 0, "pkt_pos": "48", "pkt_size": "6615", "pict_type": "I",
"side_data_list": [
{ }
] },
{ "key_frame": 0, "pts": 512, "pts_time": "0.040000", "pkt_dts": 512, "pkt_pos": "10747", "pkt_size": "1193", "pict_type": "B" },
{ "key_frame": 0, "pts": 1024, "pts_time": "0.080000", "pkt_dts": 1024, "pkt_pos": "6663", "pkt_size": "4084", "pict_type": "P" },
{ "key_frame": 0, "pts": 1536, "pts_time": "0.120000", "pkt_dts": 1536, "pkt_pos": "14848", "pkt_size": "2247", "pict_type": "B" },
{ "key_frame": 0, "pts": 2048, "pts_time": "0.160000", "pkt_dts": 2048, "pkt_pos": "17095", "pkt_size": "1971", "pict_type": "B" },
{ "key_frame": 0, "pts": 2560, "pts_time": "0.200000", "pkt_dts": 2560, "pkt_pos": "11940", "pkt_size": "2908", "pict_type": "P" },
{ "key_frame": 0, "pts": 3072, "pts_time": "0.240000", "pkt_dts": 3072, "pkt_pos": "21651", "pkt_size": "2445", "pict_type": "B" },
{ "key_frame": 0, "pts": 3584, "pts_time": "0.280000", "pkt_dts": 3584, "pkt_pos": "24096", "pkt_size": "2097", "pict_type": "B" },
{ "key_frame": 0, "pts": 4096, "pts_time": "0.320000", "pkt_dts": 4096, "pkt_pos": "19066", "pkt_size": "2585", "pict_type": "P" },
{ "key_frame": 0, "pts": 4608, "pts_time": "0.360000", "pkt_dts": 4608, "pkt_pos": "30169", "pkt_size": "2430", "pict_type": "B" },
{ "key_frame": 0, "pts": 5120, "pts_time": "0.400000", "pkt_dts": 5120, "pkt_pos": "32599", "pkt_size": "1438", "pict_type": "B" },
{ "key_frame": 0, "pts": 5632, "pts_time": "0.440000", "pkt_dts": 5632, "pkt_pos": "26193", "pkt_size": "3976", "pict_type": "P" },
{ "key_frame": 0, "pts": 6144, "pts_time": "0.480000", "pkt_dts": 6144, "pkt_pos": "36043", "pkt_size": "2339", "pict_type": "B" },
{ "key_frame": 0, "pts": 6656, "pts_time": "0.520000", "pkt_pos": "38382", "pkt_size": "1442", "pict_type": "B" },
{ "key_frame": 0, "pts": 7168, "pts_time": "0.560000", "pkt_pos": "34037", "pkt_size": "2006", "pict_type": "P" }
Steps to reproduce
Run the videoio_encapsulate.write test on video/sample_322x242_15frames.yuv420p.libx264.mp4 and check the output in ffprobe.
ffprobe ENCAPSULATED_VIDEO -select_streams v:0 -of json=compact=1 -show_entries frame=pts,pts_time,pkt_dts,pkt_pos,pkt_size,key_frame,pict_type
Issue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- I updated to the latest OpenCV version and the issue is still there
- There is reproducer code and related data files (videos, images, onnx, etc)