Decode sparse video keyframes, reconstruct smooth seek previews with 2D FFT spatial blur and 1D FFT temporal interpolation, entirely on the client-side. Includes both an MP4 keyframe demo and a WebRTC loopback demo for sparse preview streaming.
-
Local MP4 demo: MP4Box demuxes a local MP4, extracts keyframes, and feeds them into the blur + interpolation pipeline.
-
WebRTC loopback demo: Encoded keyframes are sent through a local DataChannel loopback, then reconstructed on the receiver with the same blur + interpolation pipeline.
-
Spatial blur (2D FFT): Each keyframe's Y-plane (grayscale) is downscaled to a power-of-2 preview size, transformed with a 2D FFT, lowpass-filtered (keeping ~15% of coefficients), and reconstructed via inverse FFT into a blurry but recognizable preview.
-
Temporal interpolation (1D FFT + Phase Shift Theorem): Batches of keyframes are transformed along the time axis per-pixel. High-frequency temporal bins are discarded, and synthetic in-between frames are generated at fractional time offsets, producing multiplier x more frames than were actually decoded (default 4x).
The broader target is live streaming and WebRTC: send only keyframes, or a sparse subset of encoded chunks, and let the client fill in the gaps with FFT-based interpolation.
Grayscale only, power-of-2 dimensions, and intentionally low-resolution preview output.
Color (3x FFTs for YCbCr/RGB channels), denoising, and maybe higher resolution.
- Inspired by kyndinfo's article on Fourier Series;
- Using MP4Box for MP4 demuxing.