Please note this comment from here:

Actually there are two "processing" formats:
backend SDK -> (1) -> OpenCV -> (2) -> Application

Related properties:

• CAP_PROP_FOURCC usually works with (1). V4L stores this value in capture->palette
• CAP_PROP_FORMAT works with (2) "Format of the %Mat objects returned by VideoCapture::retrieve()"
• CV_CAP_PROP_CONVERT_RGB is early version of CAP_PROP_FORMAT. Some backends support only one property and don't support another, some of them support both.

(1) is capture/camera "native" format. Like MJPEG from here:

$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'MJPG' (compressed)
	Name        : Motion-JPEG

cv::VideoCapture is able to take this MJPEG stream and then converts it into BGR frames (also it is able to pass it "as is" via CV_8UC1 buffer).

yes, I am aware of that. However for (2) there is also cvtColor. So the context here is "just" optimization as in increasing throughput.

There I argue that e.g. CV_CAP_PROP_MODE is the wrong approach. Instead the interface should be more low-level to properly expose the hardware details.
The reason is that you can not consider (1) and (2) separate, but (2) also affects (1).

For instance in the MSMF backend: CV_CAP_MODE_YUYV > MFVideoFormat_YUY2. But for most cameras MFVideoFormat_MJPG (+decoding) would increase the frame-rate as the USB2 throughput is the bottleneck.

My point here is that the VideoCapture cannot know these details as they are hardware-depended and we should expose that to the user (as in allow (1) == (2)).

Therefore I consider CAP_PROP_FORMAT an inferior approach to CV_CAP_PROP_CONVERT_RGB. If anything I would replace it by something like: #5485 (comment)

However this API does not exist yet, but maybe we can add it during the 4.0 cycle. Having to keep VideoCaptureModes as well would complicate that.