What if every badminton player had a world-class coach watching their every move — reacting in under 500ms?

RallyCoach is a full-stack AI coaching platform that transforms how athletes train. By fusing real-time computer vision with Google Gemini 3's structured reasoning, RallyCoach delivers live commentary, transparent corrections, and personalized training plans — all from a webcam.

No hardware. No sensors. Just you, your racket, and an AI coach that explains its thinking.

RallyCoach isn't a thin wrapper around an LLM. It's an end-to-end coaching system that exploits three Gemini 3 breakthroughs:

Gemini 3 doesn't just say "raise your elbow" — it reasons why. Each coaching cue arrives with a reason field exposing the model's internal logic:

{
  "cue": "Elbow higher on backswing",
  "commentary": "Your elbow is dropping to 128° — you're losing 30% of your smash power!",
  "reason": "elbow_angle=128° is below the 150° threshold for overhead shots, reducing kinetic chain efficiency",
  "focus_metric": "elbow_angle",
  "is_green": false,
  "latency_ms": 340
}

Players don't just get corrected — they understand biomechanics. That's coaching, not autocorrect.

At sub-500ms latency, Gemini 3 Flash delivers coaching cues fast enough for active movement correction. We measure and display latency per-cue so users can see the AI responding in real time. This isn't post-hoc analysis — it's a coach watching you swing.

Gemini 3's ResponseSchema validation ensures every response is machine-parseable JSON — no regex gymnastics, no prompt-and-pray. Combined with configurable thinkingBudget, we control the reasoning depth per use case:

• Live webcam pose tracking via MediaPipe (33-point skeleton, entirely client-side)
• Green/red skeleton overlay — instant visual feedback on form quality
• Gemini 3 Flash micro-coaching every 1-2 seconds with commentary + reasoning
• 5 biomechanical rules: elbow angle (overhead/drive), knee bend, stance width, body rotation
• Drill-specific form checks (smash drill vs. net play vs. footwork)
• Session recording to Supabase cloud for later review
• AI Insights sidebar showing live commentary stream and latency metrics

• Upload any badminton video — no special camera required
• Multi-frame pose extraction with aggregated metrics
• Gemini 3 Pro deep analysis: top issues (severity-ranked), targeted drills with step-by-step instructions, 5-day personalized training plan
• Auto skill-level detection (Beginner / Intermediate / Advanced)
• Optional drill visualization generated by Gemini 3 Pro Image
• Multi-player detection for doubles analysis

• Court homography mapping normalizes video coordinates to BWF court space
• Shot classification: clear, drop, smash, drive, net, serve, lift
• 2D/3D trajectory visualization with Three.js + React Three Fiber
• Deterministic tactical recommendations with confidence scoring
• Position heatmap analysis to identify opponent weak zones

• AI-powered equipment matching based on skill level + specific weaknesses
• 15+ racket catalog (Yonex, Li-Ning, Victor) with detailed specs
• Match scoring (0-100) with reasoning for each recommendation
• eBay purchase links for instant buying
• Favorites system synced to user profile

• Progress visualization with Recharts (session trends, improvement tracking)
• Detailed session replay with skeleton overlay, metrics timeline, form scores
• Aggregate statistics across all sessions

┌─────────────────────────────────────────────────────┐
│                    Browser (Client)                  │
│  ┌──────────┐  ┌──────────┐  ┌───────────────────┐  │
│  │ MediaPipe│  │  Canvas   │  │   React Three     │  │
│  │ Pose     │→ │  Skeleton │  │   Fiber (3D)      │  │
│  │ (WASM)   │  │  Overlay  │  │   Court View      │  │
│  └──────────┘  └──────────┘  └───────────────────┘  │
│       ↓ pose metrics (33 landmarks per frame)        │
└──────────────────────┬──────────────────────────────┘
                       │ JSON (every 1-2s)
                       ▼
┌─────────────────────────────────────────────────────┐
│              Next.js 14 API Routes                   │
│  ┌────────────────┐  ┌────────────────────────────┐  │
│  │ Rules Engine   │  │ Gemini 3 Flash / Pro       │  │
│  │ (local eval)   │→ │ Structured JSON output     │  │
│  │ 5 form rules   │  │ + thinking tokens          │  │
│  └────────────────┘  └────────────────────────────┘  │
│       ↓ coaching cues + commentary + reasoning       │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│                   Supabase                           │
│  ┌────────┐  ┌──────────┐  ┌───────────────────┐    │
│  │  Auth   │  │ Postgres │  │ Storage (videos)  │    │
│  │  (JWT)  │  │ sessions │  │ TUS resumable     │    │
│  │         │  │ analysis │  │ upload            │    │
│  └────────┘  └──────────┘  └───────────────────┘    │
└─────────────────────────────────────────────────────┘

Key design decisions:

• Hybrid AI architecture: Local rules engine provides instant validation; Gemini 3 adds nuanced, contextual coaching
• Client-side pose detection: Zero server load for video processing — MediaPipe WASM runs entirely in-browser
• Rate-limited AI calls: 2-second minimum between Gemini requests to prevent token waste without sacrificing responsiveness
• Graceful degradation: If Gemini API fails, cached responses keep the session running

RallyCoach/
├── backend/                    # Python FastAPI server
│   ├── main.py                 # API endpoints
│   ├── analysis.py             # Video analysis pipeline
│   ├── pose_estimation.py      # Server-side MediaPipe
│   ├── gemini_client.py        # Gemini 3 Flash integration
│   ├── video_annotator.py      # Skeleton overlay rendering
│   └── requirements.txt
├── rallycoach/                 # Next.js 14 frontend
│   ├── src/
│   │   ├── app/
│   │   │   ├── page.tsx              # Landing page
│   │   │   ├── dashboard/            # Progress dashboard
│   │   │   ├── analytics/            # Video analysis
│   │   │   ├── practice/             # Real-time coaching
│   │   │   ├── strategy/             # Tactical analysis
│   │   │   ├── racket/               # Equipment finder
│   │   │   ├── history/              # Session history
│   │   │   └── api/                  # Route handlers
│   │   │       ├── practice/tick/    # Gemini 3 Flash live cues
│   │   │       ├── analysis/start/   # Gemini 3 Pro analysis
│   │   │       ├── strategy/         # Tactical engine
│   │   │       └── racket/           # AI recommendations
│   │   ├── components/               # React components
│   │   │   ├── PoseSandbox3D.tsx     # 3D skeleton viewer
│   │   │   ├── CourtTrajectory3D.tsx # 3D court visualization
│   │   │   └── DashboardNav.tsx      # Navigation
│   │   └── lib/
│   │       ├── gemini.ts             # Gemini 3 client
│   │       ├── rules-engine.ts       # Form evaluation (5 rules)
│   │       ├── pose-utils.ts         # Angle calculations
│   │       ├── scoring-rules.ts      # Banded scoring system
│   │       ├── strategy_engine/      # Rally analysis
│   │       └── courtSpec.ts          # BWF court specs
│   └── package.json
├── supabase-setup.sql                # Database schema
└── supabase-migration-v2.sql         # Extended schema

• Node.js 18+ / Python 3.11+
• Supabase account
• Google AI Studio API key (Gemini 3 access)

# Create Supabase project, then run in SQL Editor:
# → supabase-setup.sql (base tables)
# → supabase-migration-v2.sql (extended schema)
# Create a Storage bucket named "videos"

cd backend
python -m venv venv && venv\Scripts\activate  # Windows
pip install -r requirements.txt
# Configure .env: GEMINI_API_KEY, SUPABASE_URL, SUPABASE_ANON_KEY, SUPABASE_SERVICE_KEY
python main.py  # → http://localhost:8000

cd rallycoach
npm install
# Configure .env.local: NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, GEMINI_API_KEY
npm run dev  # → http://localhost:3000

RallyCoach evaluates form against 5 research-backed biomechanical rules:

Each frame is evaluated against active rules, producing a green/red classification displayed as the skeleton overlay color. The pass/fail ratio across a session becomes the player's form score.

• Google Gemini 3 — Flash, Pro, and Pro Image models
• MediaPipe — Real-time pose estimation
• Next.js 14 — React framework with App Router
• Three.js — 3D court visualization
• Supabase — Auth, database, and storage
• FastAPI — Python backend
• OpenCV — Video processing

MIT License