FlexiHand

Low commitment and quick tool to get your hand health a kick start!

Comment

Inspiration

Hand mobility and health are often overlooked, especially among software engineers and computer science students who spend long hours typing. Repetitive strain, reduced mobility, and poor hand health can develop gradually without users realizing it until discomfort or injury appears.

We wanted to create a tool that helps people become aware of their hand mobility in a simple, interactive way.

What it does

FlexiHand is a web application that evaluates a user’s hand mobility through a series of five guided tests. During the evaluation, the app uses the user’s webcam to track their hand movements in real time. It then visually overlays a hand skeleton so users can see how their hand is being tracked. Each test analyzes specific aspects of hand mobility, such as finger extension, flexion, and positioning, and after completing all five tests, the app calculates a mobility score out of 100%.

Once the evaluation is complete, the user is taken to the Progress tab, where they are recommended hand exercises based on their performance. These exercises help users improve flexibility, mobility, and overall hand health.

How we built it

FlexiHand uses computer vision and custom gesture analysis to evaluate hand mobility. We built the frontend using HTML, CSS, and JavaScript, and integrated MediaPipe’s Hand Landmarker to detect and track 21 hand landmarks in real time using the user’s webcam.

The system works as follows:

  • MediaPipe detects the hand and returns landmark coordinates.
  • A skeleton overlay is rendered on the screen to visualize the tracked hand.
  • The landmark data is sent to a gesture processing module, which calculates distances and joint angles between key points.
  • These values are normalized to ensure consistency across different hand sizes and camera distances.
  • The processed data is then compared against predefined reference gestures stored in a separate JavaScript module.
  • Based on how closely the user’s movements match the reference gestures, the system calculates a mobility score.
  • The app then recommends appropriate exercises to help improve weak or limited mobility areas.

This architecture allowed us to separate detection, processing, comparison, and recommendation logic, making the system easier to build and extend.

Challenges we ran into

One of our main challenges was properly integrating Firestore for storing user progress. Due to several issues during the limited hackathon timeframe, we were unable to fully implement cloud storage. As a result, we adapted by storing user progress locally, which ensures the core functionality of tracking results reliably.

Another significant challenge was determining how to accurately evaluate hand mobility using landmark data. We had to manually define which distances and joint angles between landmarks were most meaningful for assessing specific hand movements. This required experimentation, testing different combinations, and fine-tuning calculations to produce realistic and useful mobility scores.

We also faced the challenge of normalization. Because every user has different hand sizes and distances from the camera, raw measurements alone were inconsistent. To solve this, we implemented normalization techniques so that mobility evaluations were based on relative proportions rather than absolute distances. This ensured the scoring system remained fair and consistent across different users.

Accomplishments that we're proud of

One accomplishment we’re especially proud of was successfully implementing Firebase Authentication. Users can securely create accounts, log in, and reset their passwords. The password reset feature sends a real email to the user’s Gmail account, making the app feel complete and production-ready.

We’re also proud of improving the quality and stability of the hand tracking visualization. Initially, the hand skeleton appeared jittery due to small frame-to-frame detection variations. To solve this, we implemented a moving average filter that tracks landmark positions over the past five frames and calculates the average position. This significantly smoothed the skeleton overlay, resulting in more stable tracking and more accurate gesture analysis.

What we learned

Throughout this project, we learned how to integrate Firebase Authentication into a web application, including implementing secure login, account management, and password reset functionality using real email verification. This gave us valuable experience working with real-world authentication systems.

We also learned the importance of normalization when working with computer vision data. Because users have different hand sizes and positions relative to the camera, raw measurements alone were unreliable. Learning how to normalize distances and angles allowed us to create a fair and consistent evaluation system across different users.

Another key learning experience was improving the stability of real-time tracking. We implemented a moving average filter to smooth landmark positions across multiple frames, which significantly reduced jitter and improved both visual quality and calculation accuracy.

In addition to the technical aspects, we learned about hand health and mobility, including exercises that help improve flexibility, strength, and reduce strain caused by prolonged computer use.

What's next for FlexiHand

Our next priority is to integrate full cloud storage using Firestore so that user progress, mobility scores, and exercise history can be securely saved and accessed across devices. This will allow users to track their improvement over time and create a more complete and persistent experience.

We also plan to expand the model to include both hands. Currently, FlexiHand is optimized for the right hand, so we want to implement left-hand compatibility and eventually support simultaneous tracking of both hands. This will make the application more inclusive and enable more comprehensive mobility assessments.

In addition, we want to enhance FlexiHand’s role as a health support tool by helping users connect with medical professionals. One feature we plan to implement is a map that shows nearby hand mobility specialists, such as physiotherapists and occupational therapists, so users can seek expert guidance when needed.

UPDATE

We managed to get Firestore to work 15 minutes before the submission deadline!

Built With

Share this project:

Updates