This repository contains the official implementation of the paper DynVFX: Augmenting Real Videos with Dynamic Content

DynVFX augments real-world videos with new dynamic content described by a simple user-provided text instruction. The framework automatically infers where the synthesized content should appear, how it should move, and how it should harmonize at the pixel level with the scene, without requiring any additional user input. The key idea is to selectively extend the attention mechanism in a pre-trained text-to-video diffusion model, enforcing the generation to be content-aware of existing scene elements (anchors) from the original video. This allows the model to generate content that naturally interacts with the environment, producing complex and realistic video edits in a fully automated way.

For more, visit the project webpage.

conda create -n dynvfx python=3.12
conda deactivate
conda activate dynvfx

pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

git clone --recursive https://github.com/DanahYatim/dynvfx.git
cd dynvfx

If you already cloned without --recursive:

git submodule update --init --recursive

pip install -r requirements.txt

cd third_party/evfsam2
pip install -r requirements.txt
cd model/segment_anything_2
python setup.py build_ext --inplace
cd ../../..

This repository uses OpenAI's GPT-4o as the VFX Assistant. Create an API key at OpenAI Platform.

Save your key in vfx_assistant/.env:

OPENAI_API_KEY=<your_key>

# 1. Prepare your video frames (720x480, 49 frames at 8fps)
ffmpeg -i input.mp4 -vf "scale=720:480,fps=8" data/my_video/%05d.png

# 2. Edit configs/user_config.yaml with your paths and desired content

# 3. Run inversion to extract refference keys and values
python inversion.py --user_config_path configs/user_config.yaml

# 4. Run DynVFX
python run.py --user_config_path configs/user_config.yaml

Edit configs/user_config.yaml with the following parameters:

See Tips section for configuration options.

Your input video should be provided as individual frames in a directory:

data/input_frames/
├── 00000.png
...
├── 00048.png

The method works best with:

• Resolution: 720×480
• Frame rate: 8 fps
• Frame count: 49 frames (~6 seconds)

Resize the video and extract the frames:

ffmpeg -i input.mp4 -vf "scale=720:480,fps=8" data/my_video/original/%05d.png

To extract the reference keys and values, we first obtain the intermediate latents by inverting the input video:

python inversion.py --user_config_path configs/user_config.yaml

Configuration - Make sure video_path and latents_path are set in your user_config.yaml file.

Note:

• 🔬 For paper comparison: This step is REQUIRED
• 🎯 For best quality: Run inversion for optimal scene alignment
• 🎲 For quick testing: Can be skipped, but results may drift

The pipeline consists of three stages:

1. 🤖 VFX Assistant — GPT-4o interprets the edit instruction and generates captions
2. 🎭 Text-based Segmentation — EVF-SAM extracts masks of scene elements
3. 🎬 DynVFX Pipeline — Iterative refinement with AnchorExtAttn


Run the entire pipeline in one command:

# In configs/user_config.yaml
mode: "auto"

python run.py --user_config_path configs/user_config.yaml

**Stage 1: Generate 🤖 VFX Assistant + EVF-SAM outputs and review

# In configs/user_config.yaml
mode: "generate"

python run.py --user_config_path configs/user_config.yaml

👀 Review the generated protocol at output_path/output_for_vfx_protocol.json and masks in masks_dir.

Stage 2: Execute with the approved protocol

# In configs/user_config.yaml
mode: "execute"

python run.py --user_config_path configs/user_config.yaml

dynvfx/
├── configs/
│   ├── base_config.yaml      # Pipeline hyperparameters
│   ├── user_config.yaml      # User-specific settings
│   └── inversion_config.yaml # Inversion settings
├── models/
│   ├── get_masks_from_sam.py # SAM mask generation
│   └── get_source_mask.py    # Source mask extraction
├── utilities/
│   ├── attention_utils.py    # Extended attention modules
│   ├── masking_utils.py      # Mask processing utilities
│   └── utils.py              # General utilities
├── vfx_assistant/
│   ├── protocol.py           # VFX Assistant (GPT-4o)
│   ├── system_prompts.py     # System prompts
│   └── .env                  # API keys (create this)
├── third_party/
│   └── evfsam2/              # EVF-SAM installation
├── dynvfx_pipeline.py        # Main pipeline
├── inversion.py              # DDIM inversion
├── run.py                    # Entry point
└── requirements.txt          # Dependencies

Enable logging to save intermediate results:

# In configs/base_config.yaml
with_logger: True

This saves to output_path:

• Input video and source masks
• Intermediate samples and target masks
• Latent mask visualizations

This work builds on:

• EVF-SAM - Base text-prompted segmentation model
• CogVideoX-5B - Base text-to-video model
• ChatGPT - Base visual language model

If you use this work, please cite:

@misc{yatim2025dynvfxaugmentingrealvideos,
      title={DynVFX: Augmenting Real Videos with Dynamic Content}, 
      author={Danah Yatim and Rafail Fridman and Omer Bar-Tal and Tali Dekel},
      year={2025},
      eprint={2502.03621},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.03621}, 
}