RukAI

Ruka AI, your trusted jump rope coach

Comment

RukAI - The Multimodal AI Jump Rope Coach

RukAI is a real-time, multimodal fitness agent that watches, analyzes, and coaches your jump rope form using continuous video and audio. Built for the Gemini Live Agents hackathon track, it moves beyond text interactions by acting as a true physical-world companion.

❁The Vision & Track Alignment

Traditional fitness apps just count reps; RukAI actually sees you. By processing a 30fps webcam feed, it tracks your biomechanics in real-time. If your form breaks down, the AI instantly interrupts your workout with live, spoken audio corrections to fix your technique before injury.

  • Real-time Interaction: Processes continuous video and audio.
  • Interruptible Coaching: Pushes real-time heuristic flags via WebSocket to trigger Gemini Live form corrections out loud.
  • Mandatory Tech: Powered by the Gemini Live API, hosted on Google Cloud/Firebase, and utilizes Google's ADK for post-workout analytics.

❁ Key Features

  • Live Audio Feedback Loop: Streams generative coaching audio via WebSockets for real-time interventions.
  • Dynamic Training Dashboard: Features a dot-matrix activity heatmap, custom regimens, and smart SVG streak tracking.
  • ⚠️ Disclaimer (The "Invisible Rope" Exception): RukAI currently operates as a biomechanical proof-of-concept. The vision engine utilizes MediaPipe heuristics to track the user's body mechanics (jumping cadence, elbow flares) rather than the physical jump rope itself. This acts as a permissive "testing mode," allowing judges to trigger the Gemini Live audio loop right from their desks without needing a physical rope.

❁ Architecture & Tech Stack

  • Frontend: React.js, HTML5 Canvas, Web Audio API
  • Backend: Node.js, WebSockets (ws)
  • AI & Vision: Gemini Live API, ADK, Google MediaPipe Tasks Vision
  • Cloud & DB: Firebase Hosting, Firestore, Google Cloud (OAuth 2.0)

❁Setup Instructions

Prerequisites

  • Node.js (v16+)
  • Firebase Project (Firestore enabled)
  • Google Cloud Project (OAuth 2.0 Client IDs configured)

Running Locally

  1. Clone the repository: git clone [https://github.com/mikitoxo/RukAI.git](https://github.com/mikitoxo/RukAI.git) cd RukAI
  2. Install Frontend Dependencies: cd frontend npm install
  3. Install Backend Dependencies: cd backend npm install
  4. Seed the Database: Ensure your Firebase serviceAccountKey.json is in the backend folder, then run: node seedRegimens.js
  5. Run the App: Start the backend socket server: node server.js Start the React frontend (in a new terminal): npm start

❁ Future Scope

  • Dedicated Computer Vision AI: Replacing heuristic testing with a custom-trained ML model optimized for jump rope biomechanics and maneuvers (like crossovers).
  • Native Mobile Port: Wrapping the application in React Native for offline-first iOS/Android support.

Built With

Share this project:

Updates