Inspiration
Stroke and brain injuries can be debilitating, often affecting a person’s ability to move and function independently.
This project was inspired by seeing recovery happen up close and realizing how much of it depends on consistent physical therapy. While clinical sessions provide structure, that support often disappears once patients return home.
A major part of recovery is repetition and consistency, but at-home therapy is often unstructured and lacks feedback. That gap in guidance is what motivated me to build Mirror, a way to bring more support into the recovery process outside of the clinic.
What it does
Mirror is a computer vision–powered recovery coach that helps patients perform rehabilitation exercises at home with real-time tracking and feedback.
It guides users through facial, arm, and hand exercises, analyzes movement quality, and provides instant feedback to help ensure exercises are performed correctly. Mirror also offers guided sessions with voice coaching, making recovery more structured, accessible, and engaging outside of traditional therapy settings.
How I built it
Mirror is built as a web application using:
- React + Vite for the frontend UI and real-time interaction
- MediaPipe (Face, Pose, and Hand Landmarkers) for computer vision tracking
- Custom metric algorithms to calculate movement quality, symmetry, strength, and stability Rule-based evaluation logic to determine correctness and generate feedback
- FastAPI backend to securely handle external API calls
- ElevenLabs for voice-based coaching and guidance
The system processes webcam input in real time, extracts body landmarks, computes movement metrics, and translates them into actionable feedback for the user.
Challenges I ran into
One of the biggest challenges was translating raw computer vision data into meaningful rehabilitation feedback. Detecting movement is relatively straightforward, but determining whether an exercise is performed correctly required careful calibration, normalization across different users, and well-tuned thresholds.
To address this, I built custom metric systems for each exercise, such as symmetry for facial movements and range of motion for arm and hand exercises. These metrics were normalized based on body proportions and calibrated in real time so the system could adapt to each user. I also used hold timers and rule-based logic to make feedback more stable and reliable instead of reacting to single-frame changes.
Another challenge was synchronizing guidance with user actions. Since the experience is voice-driven, I had to ensure that audio prompts, movement tracking, and UI feedback all felt aligned. This required structuring exercises into clear states and coordinating when feedback is triggered so the experience feels smooth rather than overwhelming.
I also explored integrating AI-generated feedback using Gemini, but was not able to fully implement it within the time constraints. Instead, I focused on building a reliable rule-based feedback system, with plans to incorporate more adaptive AI guidance in the future.
Accomplishments I'm proud of
I’m proud of building a fully functional end-to-end system that goes beyond a prototype and actually guides users through rehabilitation exercises in real time. Mirror is able to track facial, arm, and hand movements, analyze performance, and provide immediate feedback, all within a single web application.
I’m also proud of designing a consistent system across different types of exercises. Even though each movement is different, I was able to create a shared pipeline of tracking, metric computation, and feedback that scales across use cases.
Finally, I’m proud of designing the experience to be accessible and easy to use. The voice guidance allows users to move through exercises without relying on complex navigation, which is especially important for those with limited mobility. Combined with a simple interface and structured sessions, the app feels supportive and usable rather than overwhelming.
What I learned
I learned how to bridge the gap between raw computer vision outputs and meaningful user feedback. Working with MediaPipe showed me that detecting landmarks is only the first step, and that turning those into useful metrics requires careful thinking about normalization, calibration, and thresholds.
I also learned how to design real-time systems that remain responsive while continuously processing data. Managing state, timing, and updates without overwhelming the UI was an important challenge.
Beyond the technical side, I gained a better understanding of user experience design, especially for accessibility. Keeping interactions simple, clear, and supportive is critical when building tools for recovery and rehabilitation.
What's next for Mirror
The next step is to make Mirror more adaptive and personalized. I plan to integrate AI-generated feedback to provide more natural, context-aware coaching instead of relying only on rule-based logic.
I also want to expand the range of exercises and improve the accuracy of movement evaluation across different users and environments. Adding persistence and session tracking will allow the system to store past sessions, track key metrics over time, and provide clear evaluations of user improvement. This will also enable more adaptive sessions that adjust based on a user’s progress, helping make recovery more personalized and consistent.
Log in or sign up for Devpost to join the conversation.