System Information

OpenCV version 4.8.1
OS: Windows 10
Python 3.12.something

Detailed description

Reading a video with params cv.CAP_PROP_FORMAT, -1, the packets/frames are emitted in file order (increasing offset or DTS), not logical order (increasing PTS). That would be acceptable but...

• CAP_PROP_POS_FRAMES is counted in order of these packets occurring in the file, not in logical (PTS) order
• CAP_PROP_POS_MSEC is always zero

This makes it difficult for me to identify the PTSes of keyframes that can be seeked to quickly.

Modern files contain B frames, which can reference future frames. This usually requires P and I frames to be encoded before those, so they'd be decoded and ready when such a B frame needs to be decoded.

get(CAP_PROP_LRF_HAS_KEY_FRAME) correctly reports a frame/packet being a keyframe. That is how I noticed this initially. The first keyframe correctly was at POS 0 but the second one should have been at POS 250 when it was reported to be at POS 246.

Steps to reproduce

import sys
import cv2 as cv

[filepath] = sys.argv[1:]
vid = cv.VideoCapture(filepath, apiPreference=cv.CAP_FFMPEG, params=[
    cv.CAP_PROP_FORMAT, -1, # RAW
])
assert vid.isOpened()
while True:
    index_before = int(vid.get(cv.CAP_PROP_POS_FRAMES))
    rv = vid.grab()
    if not rv: break
    (rv, data) = vid.retrieve()
    if not rv: break
    is_keyframe = vid.get(cv.CAP_PROP_LRF_HAS_KEY_FRAME) != 0
    pos_sec = int(vid.get(cv.CAP_PROP_POS_MSEC)) * 1e-3
    print(f"{index_before:4d}", f"{pos_sec:.3f}", is_keyframe, data.shape)

Compare to ffprobe:

ffprobe the_file.mov -select_streams v:0 -of json=compact=1 -show_entries frame=pts,pts_time,pkt_pos,pkt_size,key_frame,pict_type 2>/dev/null

ffmpeg reports the frames in increasing PTS order. Compare packet sizes when ordered by pkt_pos instead, and they'll match what OpenCV emits.

Since the issue form refuses to accept both .mkv and .mov (which it says is ok), I've uploaded a test file here: https://drive.google.com/file/d/1H60MLH_8zz7ue9f8emYOows7_JS71woS/view?usp=sharing

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)