USign - Live ASL Translator

People have a difficult time communicating with deaf or mute individuals. Our solution provides a live translation from ASL to English, with audio for real-time conversation.

Comment

Inspiration

We observed the difficulties of deaf and/or mute people communicating with others. Most people are not familiar with, much less fluent, in ASL. We wanted to make a bridge for communication between those who speak with their hands, and those who speak with their voice.

What it does

It takes a video or live stream of ASL signing, and using Gemini to process the frames sequentially, it compares with a MongoDB database of known signs, and outputs text and/or voice (With ElevenLabs) for a quick, live translation.

How we built it

We used Gemini to analyze frames from a video source (webcam) and output descriptions, which it compared to known signs in a MongoDB database. The program would then output the text translation, with a confidence rating based on how well it lines up with the known sign description, and ElevenLabs software takes the text and outputs it audibly as speech.

Challenges we ran into

Our biggest problems were related to AI. Getting the AI to process each frame quickly enough to output a reliably accurate description, then compare it to the database before the next frame was difficult. The limits on Google's ai studio were also very difficult to work around, as we constantly ran into quota limits.

Accomplishments that we're proud of

The app in its current state can take a live feed of video and audibly output the English ASL translation from a MongoDB database with sign descriptions. All implemented GUI functions work as intended, and we're happy with the web design. We successfully launched the app through DigitalOcean and got it to run on more than one device without need for a terminal/shell.

What we learned

How to connect different frameworks; the database, AI, and web host. We also learned about how to get the full stack to work together- frontend and backend.

What's next for USign

We plan to continually expand the vocabulary so that all ASL signs can be accurately translated, and would like to integrate it into AR/VR devices like Meta Glasses or VR Headsets for live translation in social apps or text display in real-world conversations. Optimization of the AI processing is a tough challenge we will continually work to improve.

Share this project:

Updates