Rosetta Sign

Inspiration

Language is the most powerful source of community. In today's globalized world, countless programs and tools like Rosetta Stone and Duolingo help bring communities together by helping expand a language's reach. But what about American Sign Language? Our team sought to create a bridge for this gap.

What it does

Our program utilizes a python-based OpenCV framework to capture video in real time. Using a series of image processing, we are able to isolate components of the hand and pinpoint edges on fingers. We then feed this through a neural network that's trained on the ASL alphabet. Currently, the program tries to identify the letter the user is signing, and prints it to the console.

How I built it

To build this project, we relied heavily on OpenCV libraries to perform much of the recognition and detection. We applied our custom processing in order to better fit the module's abilities.

Challenges I ran into

In the limited time span of a hackathon, it's difficult to procure a well-trained, non-biased model. Our model isn't very accurate, but presents a step forward in the right direction.

Accomplishments that I'm proud of

Fantastic edge detection and display in the processed image.

What I learned

Machine learning requires extensive and dedicated training to actually be as effective as people think it is.

What's next for Rosetta Sign

Our original intent was to create a tutoring app that can pose a letter or gesture to the user, recognize when the user signs a specific level, and adjusts the next level accordingly. Eventually, the user should be able to gain a decent grasp over ASL.

Built With

• computer-vision
• edge-detection
• keras
• machine-learning
• opencv
• python
• tensorflow