-
Notifications
You must be signed in to change notification settings - Fork 687
Description
Our VideoAsset archetype has no issues with b-frames at all. However, the (as of writing upcoming) VideoStream archetype won't support b-frames in its first few iterations.
B-frames for h.264/h.265 create additional complexity as they require differing decode timestamps (DTS) and presentation timestamps (PTS):
We have to add decode timestamps as a separate optional field and take care of the necessary reordering. An additional issue is that currently we assume that all video frames come in-order (rather special as all our other logging it out-of-order!). This greatly complicates access into rerun chunks containing frame data.
As encoders emit b-frames alongside both prior and successive frames, one way to solve this is to extend VideoStream to have arrays of presentation time offsets and allow several frames to be in a single sample. This way, the stream of samples is still ordered in decoding order, keeping decoder ingestion simple. Since any use of b-frames naturally causes significant latency, this solution is likely truthful to the actual output.
(Note, that none of the decoders we worked with so far actually needs the actual value of the DTS, so we're not missing this information anywhere!)
See also:
TODO: What about bframe support on AV1 and VP9? We probably support this fine already, as these codecs don't have the sample ordering issue AVC/HEVC have. But we should test it!