Dexterity

Inspiration

The inspiration for this project comes from the goal of making communication more accessible for people who use sign language. There are over 430 million people worldwide living with disabling hearing loss. Platforms like Google Meet are used every day for school and work, but they still don’t support real-time ASL to text translation. Many people face challenges when communicating across language barriers, and this project strives to bridge that gap using technology.

What it does

The system tracks hand movements and interprets them into real-time text that appears on the screen, making sign language easier to understand. This text can then be transformed into speech, allowing for smoother communication. Also, the program includes a guide named Dexter who helps users learn and perform signs more effectively. Additionally, there is a dictate feature enabling speech-to-text, allowing for 2-way communication.

How we built it

With the help of Claude, we built a website on HTML, CSS, and JavaScript which we then tested and ran on WebStorm. We also used GitHub for version control and for hosting the webpage. For the hand tracking technology, we used Google’s Mediapipe framework along with the IndexedDB API for storing the hand positioning data.

Challenges we ran into

When working with the hand tracking, we found that it was inaccurate and it lagged behind changes in hand position. We spent a lot of time ensuring that it was accurate and smooth, and eventually ended up switching from using FingerPose to MediaPipe, which made it easier to improve the quality of the handtracking. Additionally, the algorithm had preconceived notions on how the hand positions were supposed to look, interfering with the consistency in which the program registered the hand positions during testing. To fix this, we specifically trained the program using real user inputted data. This asset helps customize the experience specifically to the user's hand size, shape, etc. making it increasingly accessible to all. Additionally, the tracking data is implemented locally, and doesn’t transfer across devices. Given the timeframe of the hackathon, we weren’t able to create a database as we prioritized hosting the site on GitHub pages. However, we have plans to implement the database to share the data across devices in the future.

Accomplishments that we're proud of

Some of the accomplishments we are proud of are adding the customizability of the hand positions. This greatly improved accuracy in detecting the letters and allows accessibility for a wider audience. Another accomplishment we are proud of is our implementation of the MediaPipe framework. We were initially using the fingerpose js for detection, but the tracking was lagging behind hand movements, and the application would get confused between letters with similar hand positions (such as fists).

What we learned

We learned a lot from building Dexterity, especially how to use computer vision in real time with MediaPipe to track hand movements. We also learned how to process data in JavaScript and map it to specific ASL signs, while keeping the system fast enough to run smoothly in the browser with HTML and CSS. Another big takeaway was balancing speed with accuracy, since even small delays can affect usability. Overall, we also improved our teamwork skills by dividing tasks, debugging issues, and building a working project from scratch.

What's next for Dexterity

Our future plans for Dexterity are to add user accounts, allowing people to save and access their own trained data more easily over time. Implementing the project as a Chrome extension would allow it to run directly on platforms like Google Meet or Zoom, removing the need for a separate application and making it more convenient to use.