MediaProcessing.asCIImageSequence should produce an array of downsampled frames #223
Description
MediaProcessing.asCIImageSequence should take a visitor that can apply processing to frames and downsample (requires rendering as CIImage is just a promise!)
Right now we do this:
var videoFrameResult = await try MediaProcessing.asCIImageSequence(
video, maxFrames: maxVideoFrames, targetFPS: targetVideoFPS)
var processedFrames: [MLXArray] = []
for frame in videoFrameResult.frames {
let image =
frame
.toSRGB()
.resampled(
to: CGSize(width: fixedImageSize, height: fixedImageSize), method: .lanczos
)
.normalized(mean: config.imageMeanTuple, std: config.imageStdTuple)
.asMLXArray()
processedFrames.append(image)
}if the video is 4k, we might have 64 very large frames that we then downsample rather than downsampling as we read.