Transform audio into visual narratives. Every song deserves a screen.

CineWave is a multimodal platform that analyzes music (audio + lyrics), extracts emotional arcs and narrative structure, and generates cinematic storyboards and music video trailers using AI. It democratizes visual storytelling for independent artists who can't afford $5K–$20K production budgets.

Built for the Vultr hackathon — heavy cloud compute on Vultr; structured data in Snowflake.

Music discovery today is visual-first: TikTok, Instagram Reels, YouTube Shorts. Songs without compelling visual content struggle to reach new audiences. High-quality music videos are expensive—production crews, cinematographers, editors—and label-backed artists dominate feeds because they can produce content at scale. Creativity alone is not enough; production scale wins.

This isn’t just a video-generation problem—it’s a multimodal understanding problem. Current platforms rely on shallow metadata and genre tags. AI video tools are generic; they don’t understand emotional arcs, lyrical meaning, or sonic progression. Most treat music as background audio. CineWave treats music as structured emotional data.

Every song has an emotional trajectory: verse, chorus, bridge, build, release. From audio features, lyrics, and metadata, we can build:

• Emotional signature embeddings → “Songs that feel similar” instead of “songs that sound similar”
• Narrative mapping → Visual storytelling grounded in structure
• Emotion-aware discovery → Reels-style feed driven by feeling, not genre

CineWave combines:

1. Audio analysis — Waveform, spectral features, energy peaks, emotional arcs
2. Lyric analysis — Themes, sentiment, imagery, semantic parsing
3. Multimodal alignment — Lyrics + audio + structure → coherent narrative
4. AI visual generation — Cinematic storyboards and video via Luma (and extensible providers)
5. Reels feed — Discovery driven by generated visuals and emotional similarity

The video is the output layer; the core product is emotion-aware creative amplification.

• Upload or import — Audio files, YouTube URLs (via yt-dlp)
• Analysis pipeline — Spectrogram, features, emotional peaks (external audio-analysis service or local fallback)
• Storyboard generation — Per-shot prompts from audio features + lyrics + style preset
• Music video trailer — Multi-shot generation with Luma, continuity or hard cuts, optional LLM refinement
• Stitch & export — FFmpeg-based concatenation, audio overlay, subtitle burn
• Reels feed — Scrollable clips linked to tracks and regions
• Signature-based similarity — Global and segment embeddings for “songs similar to this moment”
• Resolution selection — 512/768/1024 (images), 720p/1080p (video)

CineWave uses Vultr for heavy cloud compute — our chosen hackathon track. Serverless (Vercel) cannot run long-running audio processing, FFmpeg, or Python ML pipelines. Vultr provides the compute tier that makes the product possible.

• Compute limits — FFmpeg, yt-dlp, librosa, and long-running analysis need real VMs, not serverless timeouts
• Docker deployment — Full stack (Next.js, worker, audio-analysis, features-api, Redis) runs via Docker Compose on a Vultr VM
• Object Storage — S3-compatible; spectrograms and audio artifacts stored in Vultr Object Storage
• Cost-effective — Dedicated CPU for predictable performance and cost

See deploy/ for Vultr setup: deploy/README.md, deploy/.env.example, Docker Compose config.

Snowflake is our analytical data warehouse. All structured outputs from analysis and generation are stored in Snowflake for reproducibility, versioning, and analytics — not just blobs in object storage.

• Structured storage — Features, peaks, transcripts, and prompts as queryable JSON (VARIANT)
• Reproducibility — Every analysis and generation is versioned and traceable
• Analytics — Query patterns, usage, and insights over time
• Hackathon alignment — Enterprise data warehouse instead of ad-hoc SQLite blobs

The ingest worker writes TRACK_ANALYSIS_EVENTS; the audio-analysis service writes to Snowflake; the app reads generations and featured reels from Snowflake when configured. See deploy/snowflake.sql, lib/snowflake-events.ts, lib/snowflake-generations.ts.

User Browser
    → Next.js (Vercel / local)
    → API Routes (upload, analyze, generate, runs, stitch)
    → Redis (BullMQ) → Ingest Worker on Vultr (ffmpeg, yt-dlp, audio-analysis)
    → Prisma + SQLite (local) / Postgres (prod)
    → Snowflake (analysis events, generations, reels)  ← data warehouse
    → Vultr Object Storage / S3 (audio, spectrograms, assets)
    → Luma (video generation)
    → FFmpeg (stitch)

Two pipelines combine metadata, lyrics, and audio/spectrogram features to produce gen AI prompts:

Pipeline 1: Features API (features/extract.py, features_app.py)

Pipeline 2: Storyboard / Run (spectrogram.py + src/lib/storyboardPrompts.ts)

• Model: OpenAI GPT-4o-mini (configurable via LLM_MODEL, default gpt-4o-mini)
• Features API: One call per song — metadata + lyric lines → global_style + per_line tags (vibe_keywords, visual_style_preset, color_palette, prompt_atoms: setting/subject/action/mood/lighting)
• Storyboard refinement: Optional second call — style/mood/story + shot count → refined { index, prompt } per shot
• Fallback: Stub/heuristic tags when OPENAI_API_KEY is missing

Extracted via librosa (STFT, n_fft=2048, hop_length=512):

• Scene plan (features pipeline): preset + vibe + setting + subject + action + mood + lighting + colors; motion_intensity from RMS at line start; chorus sections get +0.2 intensity boost
• Storyboard (run pipeline): style + energy + brightness + motifs + moment_label; optional LLM refinement for more varied descriptions

Prisma (app DB) — Users, tracks, runs, shots, jobs, assets.

Snowflake (data warehouse) — Analysis events, generations, featured reels, signatures. All structured outputs (features, peaks, transcripts, prompts) are queryable for reproducibility and analytics.

1. Track Ingest

• User uploads file or provides YouTube URL
• API creates Track, enqueues BullMQ job
• Worker (Vultr): downloads (yt-dlp) or processes file, calls audio-analysis service
• Spectrogram → Vultr Object Storage / S3; features, peaks → Snowflake (TRACK_ANALYSIS_EVENTS); Track → READY

2. Music Video Generation

• User starts run from Create or song page (track/song draft)
• POST /api/runs → getAudioFeaturesForRun → generateStoryboardPrompts → buildStoryboardAndShots
• Shots + ProviderJobs created; processRun submits Luma jobs with continuity
• Luma callback or polling updates ProviderJob status; Assets store video URLs
• POST /api/runs/[runId]/stitch concatenates clips, overlays audio, returns final video

3. Storyboard / Image Generation

• POST /api/tracks/[id]/generate or /api/generate (mock)
• Generations stored with resolution; linked to Reels for feed

• Queue: track-ingest
• Job types: processUpload, downloadYoutubeAudio
• Retries: 3 attempts, exponential backoff (2s base)
• Retention: Last 100 completed jobs
• Connection: Redis (from REDIS_URL)

• Worker: Run multiple npm run worker instances; BullMQ distributes jobs
• Redis: Single instance sufficient for ~10K jobs/day; cluster for higher throughput
• Audio Analysis / Features API: Stateless; scale behind load balancer
• Next.js: Stateless; scale behind nginx/load balancer

1. Audio analysis — CPU-bound (librosa); long tracks (5+ min) may hit timeout; consider ANALYSIS_TIMEOUT_S increase
2. Luma API — External rate limits; run orchestrator polls; continuity mode creates jobs sequentially
3. Transcription — faster-whisper is GPU-acceleratable; CPU-only on smaller VMs
4. Stitch — FFmpeg CPU-bound; large runs (10+ shots) may take 1–2 min

• Node.js 18+
• Redis (for ingest jobs)
• ffmpeg, ffprobe (for audio/video)
• yt-dlp (optional, for YouTube import)

# Install dependencies
npm install

# Copy environment
cp .env.example .env
# Edit .env: DATABASE_URL, REDIS_URL, Clerk keys

# Initialize database
npm run db:push

# Optional: seed data
npm run db:seed
# Or: npm run db:seed-songs-2025

# Terminal 1: Next.js dev server
npm run dev

# Terminal 2: Ingest worker (processes uploads, YouTube, analysis)
npm run worker

Open http://localhost:3000.

Optional — Features API (transcribe, LLM tags, scene plan):

pip install -r features_requirements.txt
uvicorn features_app:app --host 0.0.0.0 --port 8000
# Set API_BACKEND_URL=http://localhost:8000 or NEXT_PUBLIC_API_BASE_URL

Optional — Audio Analysis (spectrogram, build-prompt): see services/audio_analysis/; or use AUDIO_ANALYSIS_SERVICE_URL pointing to a deployed instance.

Prerequisites: Docker, Docker Compose; Vultr VM or any Linux host.

Vultr VM setup:

ssh root@YOUR_VULTR_IP
curl -fsSL https://get.docker.com | sh
systemctl enable docker && systemctl start docker
apt-get update && apt-get install -y docker-compose-plugin

Deploy:

cd deploy
cp .env.example .env
# Edit .env: CLERK keys, SNOWFLAKE_*, S3_*, OPENAI_API_KEY
# Run Snowflake DDL: deploy/snowflake.sql (in Snowflake worksheet)
docker compose up -d --build

Docker services (behind nginx on port 80):

Nginx routing (deploy/nginx.conf):

• / → web:3000
• /audio-analysis/* → audio-analysis:8001
• /features-api/* → features-api:8000
• client_max_body_size 50M; proxy_read_timeout 150s

Volumes: redis_data, app_data (SQLite + uploads), audio_artifacts

Health checks:

curl http://localhost/                    # Next.js
curl http://localhost/audio-analysis/health
curl http://localhost/features-api/docs   # Features API Swagger

Troubleshooting:

• Worker failing → docker compose ps; ensure Redis healthy
• Analysis timeout → increase ANALYSIS_TIMEOUT_S in .env
• DB errors → docker compose exec web npx prisma db push
• Upload 404 → check UPLOADS_DIR, app_data volume

S3 (Vultr Object Storage):

1. Create bucket in Vultr Object Storage
2. Set S3_BUCKET, S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY in .env
3. Set STORAGE_MODE=s3 for audio; spectrograms go to S3 when configured

Create run from track:

curl -X POST http://localhost:3000/api/runs \
  -H "Content-Type: application/json" \
  -H "Cookie: <auth-cookie>" \
  -d '{"trackId":"TRACK_ID","durationSeconds":15,"stylePreset":"cinematic","mode":"continuity"}'

Create run from song input (URL + metadata):

curl -X POST http://localhost:3000/api/runs \
  -H "Content-Type: application/json" \
  -H "Cookie: <auth-cookie>" \
  -d '{
    "songInput": {
      "audioSource": {"type":"URL","url":"https://example.com/track.mp3"},
      "lyrics": {"raw":"Verse lyrics..."},
      "metadata": {"title":"Song","artist":"Artist"}
    },
    "durationSeconds":12,
    "stylePreset":"noir"
  }'

Features API (upload → process → features):

curl -X POST -F "file=@song.mp3" -F "title=My Song" -F "artist=Me" http://localhost:8000/songs/upload
curl -X POST http://localhost:8000/songs/{song_id}/process
curl http://localhost:8000/songs/{song_id}/features

├── app/                    # Next.js App Router
│   ├── (landing)/          # Landing page
│   ├── (main)/             # Dashboard, create, library, discover, reels, video, storyboard, timeline
│   ├── auth/               # Clerk auth routes
│   ├── api/                # API routes (tracks, runs, providers, etc.)
│   └── demo/               # Demo pages
├── components/             # React components (landing, create, processing, ui)
├── deploy/                 # Docker, nginx, Vultr deployment
│   ├── Dockerfile.*        # web, worker, features, audio-analysis
│   ├── docker-compose.yml
│   ├── nginx.conf
│   └── snowflake.sql
├── features/               # Python features API (transcribe, LLM tags, scene plan)
│   ├── extract.py          # Main pipeline: audio → lyrics → LLM → scene_plan
│   ├── llm_tags.py         # GPT-4o-mini lyric tagging
│   ├── scene_plan.py       # Prompts from LLM + RMS
│   └── audio_analysis.py   # librosa (BPM, beats, RMS)
├── lib/                    # DB, auth, audio-analysis, storage, snowflake
├── services/
│   └── audio_analysis/     # FastAPI spectrogram service
│       ├── app.py
│       ├── analysis_runner.py
│       └── prompt_builder.py
├── spectrogram.py          # librosa STFT, peaks, bands, sections
├── src/
│   ├── lib/                # runOrchestrator, storyboardPrompts, shotPlanner, stitcher
│   ├── providers/          # Luma adapter
│   └── schemas/            # Zod schemas
├── workers/                # Ingest worker (BullMQ)
├── prisma/
│   ├── schema.prisma
│   └── seed.ts
├── scripts/                # Seed, build demo
└── server/                 # Queue definition

• Auth: Clerk handles sign-in; API routes check auth() for protected endpoints
• Uploads: File type validation (audio extensions); size limit via MAX_UPLOAD_MB
• Secrets: OPENAI_API_KEY, CLERK_SECRET_KEY, SNOWFLAKE_*, S3_* — never commit; use .env
• CORS: Next.js same-origin by default; configure for custom domains if needed
• Storage: S3 presigned URLs or public bucket; ensure bucket policy restricts writes

ISC