We offer a new perspective on approaching the task of video generation. Instead of directly synthesizing a sequence of frames, we propose to render a video by warping one static image with a generative deformation field (GenDeF). Such a pipeline enjoys three appealing advantages. First, we can sufficiently reuse a well-trained image generator to synthesize the static image (also called canonical image), alleviating the difficulty in producing a video and thereby resulting in better visual quality. Second, we can easily convert a deformation field to optical flows, making it possible to apply explicit structural regularizations for motion modeling, leading to temporally consistent results. Third, the disentanglement between content and motion allows users to process a synthesized video through processing its corresponding static image without any tuning, facilitating many applications like video editing, keypoint tracking, and video segmentation. Both qualitative and quantitative results on three common video generation benchmarks demonstrate the superiority of our GenDeF method.

• Python 3.8+
• CUDA 11.3+
• PyTorch 1.11+ and torchvision 0.12+

# Clone the repository
git clone https://github.com/aim-uofa/GenDeF.git
cd GenDeF

# Install PyTorch (adjust for your CUDA version, see https://pytorch.org/)
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 --extra-index-url https://download.pytorch.org/whl/cu113

# Install Python dependencies
pip install -r requirements.txt

# Install the project in editable mode
pip install -e .

We support training on the following video datasets:

• YouTube Driving (YTB): YouTube driving videos at 256×256 resolution
• SkyTimelapse: Sky timelapse videos at 256×256 resolution
• TaiChi-HD: Tai Chi videos at 256×256 resolution

Organize the dataset as a zip archive and place it in the data/ directory:

data/
  ytb_256.zip       # YouTube Driving dataset
  sky_256.zip       # SkyTimelapse dataset (optional)
  taichi_256.zip    # TaiChi-HD dataset (optional)

Each zip file should contain video frames organized as described in the StyleGAN-V dataset format.

Training follows a two-stage pipeline. Below we use the TaiChi-HD dataset as an example.

In this stage, we train a 2D image generator backbone with deformable convolutions. The model learns to generate single frames (i.e., num_frames_per_video=1), building a strong image generation foundation.

bash scripts/train_taichi_stage1_pretrain.sh

In this stage, we introduce the canonical image generation and deformation field prediction modules. The model learns to generate videos by warping a canonical image with a predicted deformation field. The pretrained checkpoint from Stage 1 is used as initialization.

bash scripts/train_taichi_stage2_finetune.sh

After training, generate videos using:

bash scripts/generate_videos.sh

You can customize the generation by editing the script or passing arguments directly:

python src/scripts/generate_ours.py \
    --network_pkl output/taichi_finetune/output/best.pkl \
    --num_videos 100 \
    --save_as_mp4 true \
    --fps 25 \
    --video_len 128 \
    --batch_size 25 \
    --outdir sample/taichi \
    --truncation_psi 0.9 \
    --seed 42

This codebase is built on top of StyleGAN-V. We thank the authors for their excellent work.

If you find our work useful, please consider citing:

@misc{wang2023gendef,
    title={GenDeF: Learning Generative Deformation Field for Video Generation},
    author={Wen Wang and Kecheng Zheng and Qiuyu Wang and Hao Chen and Zifan Shi and Ceyuan Yang and Yujun Shen and Chunhua Shen},
    year={2023},
    eprint={2312.04561},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}