Automate Your Health

• 

  A yoga pose that can serve as a reference

Inspiration

I know many people who need physical therapy but constantly struggle with the costs and lack of convenient options. They’re often forced to commute to another location for sessions, when they’d much rather do their exercises at home. Hearing these stories made me realize there had to be a better way—one that blends technology, accessibility, and affordability.

What it does

Automate Your Health is an AI-powered platform that helps you do your physical therapy exercises from the comfort of home. It provides:

Real-Time Feedback: Ensures you’re doing each exercise correctly. Progress Tracking: Keeps tabs on your form, reps, and improvements over time. Data Sharing: Lets you share results with coaches, doctors, or physical therapists for expert guidance and accountability. Friendly Competition: Compare stats with friends to turn your recovery into a motivating game.

How we built it

Front-End & Styling React + Vite + TypeScript for a snappy, scalable user interface. Tailwind CSS to keep the design modern and responsive without heavy CSS overhead.

AI & Computer Vision TensorFlowJS integrated into the browser for real-time pose detection. Google’s Teachable Machine used to train our model with “perfect form” data gathered from online exercise videos. This helps us detect and evaluate correct exercise form.

Google Gemini API We use the Gemini 2.0 Flash model, a multimodal model that handles live audio, video, and text. WebSockets enable a live, bidirectional connection with the Gemini API. Our current implementation leverages live bidirectional audio combined with text prompts, where we instruct the model to act as a personal trainer or physical therapist. Whenever a user’s rep counter changes, we send that data (as text) to the Gemini model, which responds verbally in real time.

Server & Integration Node.js on the back end to handle API requests, store user progress, and maintain WebSocket connections.

Challenges we ran into

Accuracy & Pose Detection: Fine-tuning the AI model to correctly identify subtle exercise movements took repeated trial and error.

Real-Time Communication: Ensuring the Gemini API and our WebSocket connections stayed efficient and error-free required careful coordination.

Privacy & Data Security: Since we track user health data, we had to ensure robust security measures and compliance with relevant standards.

Accomplishments that we're proud of

Precise AI Feedback: Our system provides accurate, timely feedback so users know immediately if their form is off.

Live Interaction: Gemini’s live audio feedback keeps the experience engaging, almost like having a personal trainer in the room.

Scalability: By using modular frameworks, our solution is set to grow with new exercises and features.

What we learned

Real-World Relevance: We discovered how important it is to design AI solutions that genuinely meet people’s everyday needs. Motivation Through Gamification: Introducing a competitive, playful element can greatly increase user engagement and commitment to therapy routines. Machine Learning Models: We learned the ins and outs of creating and fine-tuning ML models to strike the perfect balance between accuracy and efficiency.

What's next for Automate Your Health

Expanded Exercise Library: Incorporate targeted physical therapy routines for various injuries and conditions. More Models: Get more models than we can use to add a variety of trainings into our application. Deeper Gemini Integration: Enhance the multimodal capabilities, potentially including live video analysis for even more accurate feedback.

Built With

• gemini
• google'steachablemachine
• node.js
• react
• tailwindcss
• tensorflowjs
• typescript
• vite