AI-powered video editing. Click an object, apply an edit, propagate across every frame.

Built by Krish Punjabi, Bryan Lin, Justin Wang, and Daniel Zhao.

FrameShift lets you edit objects in video without frame-by-frame manual work. Upload a video, click any object, and apply edits like recolor, remove, resize, or replace — the changes automatically propagate across your entire clip.

• AI Object Detection — YOLOv11 automatically detects every object in your video. Select any object just by clicking on it.
• Background Removal — Remove or swap backgrounds instantly with AI-generated scenes.
• Color Grading — Change the color of any detected object across all frames.
• Smart Resize — Reformat for any platform in one click.

Drag and drop or upload from your device. FrameShift extracts every frame and runs object detection in the background.

Click any detected object in the editor. SAM 2 segments it precisely and tracks it across frames.

Use the toolbar to remove, recolor, resize, blur, or replace objects. Apply whole-frame effects like background removal, enhance, upscale, or restore. Or describe your edit in natural language with the AI Edit panel.

One click propagates your edit across the entire frame range. Preview the result, then render your final video.

Video Upload → FFmpeg Frame Extraction → YOLOv11 Detection
                                              ↓
              Click Object → SAM 2 Segmentation + Tracking
                                              ↓
            Apply Edit → Cloudinary Transforms (per frame)
                                              ↓
              FFmpeg Re-encode → Final Video via Cloudinary

Describe your edit in natural language (e.g., "make the person in the middle red") and FrameShift applies it.

• Node.js 18+
• Python 3.10+
• FFmpeg installed and on PATH
• Cloudinary account
• Supabase project
• Auth0 application

cd backend
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

Download the SAM 2 checkpoint:

mkdir checkpoints
curl -L -o checkpoints/sam2_hiera_tiny.pt \
  https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt

Create backend/.env:

CLOUDINARY_CLOUD_NAME=your_cloud_name
CLOUDINARY_API_KEY=your_api_key
CLOUDINARY_API_SECRET=your_api_secret
SUPABASE_URL=your_supabase_url
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key
AUTH0_DOMAIN=your_auth0_domain
AUTH0_AUDIENCE=your_auth0_audience

Start the server:

uvicorn main:app --reload

cd frontend
npm install

Create frontend/.env.local:

NEXT_PUBLIC_API_URL=http://localhost:8000
AUTH0_SECRET=your_secret
AUTH0_BASE_URL=http://localhost:3000
AUTH0_ISSUER_BASE_URL=https://your-tenant.us.auth0.com
AUTH0_CLIENT_ID=your_client_id
AUTH0_CLIENT_SECRET=your_client_secret
NEXT_PUBLIC_SUPABASE_URL=your_supabase_url
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key
NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME=your_cloud_name
NEXT_PUBLIC_CLOUDINARY_UPLOAD_PRESET=your_upload_preset

Start the dev server:

npm run dev

App runs at http://localhost:3000.