Speak2Sign

• 

  Example of character shown (Letter H)

Inspiration

The inspiration behind our project to develop a speech-to-sign language translation system using Whisper AI was the desire to bridge the communication gap between the hearing and the deaf community. We recognized the challenges faced by individuals who rely on sign language as their primary means of communication, especially when interacting with those who do not understand or know how to communicate in sign language. We aimed to create a solution that would enable seamless communication and foster inclusivity.

What it does

Speak2Sign is a revolutionary project that utilizes Whisper AI to translate speech and audio into sign language. The goal of the project is to bridge the communication gap between the hearing and the deaf community by providing an accurate and reliable speech-to-sign language translation system.

The system takes spoken language as input and uses the power of Whisper AI, an advanced language model, to analyze and interpret the speech. It then generates corresponding sign language gestures. The translated sign language is displayed on a screen or device, allowing deaf individuals to understand and communicate effectively with those who do not know sign language.

How we built it

This project uses Python, Whisper API, PIL imaging libraries, and the famous Matplotlib library. I created a folder containing all the lowercase letters within ASL (American Sign Language). Next, a list cleaning process removes all punctuation, capitalization, and spaces within the speech. Then, another process iterates throughout every word formed in the new list and creates a corresponding list holding index values from the set of letters. This set of letters matches the file name of ASL images. Next, a for loop iterates through out the list of indices and finds the matching ASL image, which is then displayed sequentially. None of these would be possible without the use of WhisperAI API, PIL image displaying libraries, and the built in functions of Matplotlib.

Challenges we ran into

There were many unexpected errors that I had discovered within this project. Since this was only my 2nd time writing code in Python, I lacked the fundamental knowledge to create simple things such as lists or for loops. Furthermore, I did not know how to import APIs. Hence, this was a large learning curve for me and I stayed up late learning and installing these dependencies. Another issue that I am currently trying to fix to clean up and make the system look cleaner. Currently, it displays all the characters individually, however it is difficult to fix this issue since each word has a different length and to individually present each one would require multiple 2D arrays.

Accomplishments that we're proud of

I was extremely happy when WhisperAI's API had first worked. Furthermore, learning how to use Matplotlib and PIL was monumental as this was my first time using these libraries and learning them.

What we learned

During the development of Speak2Sign, we learned a great deal about the intricacies of sign language, the challenges faced by the deaf community, and the importance of accurate and fluent translation. We gained a deeper understanding of the nuances of sign language gestures, facial expressions, and body movements, and how they contribute to effective communication. We also learned about the power of AI and language models like Whisper AI in bridging communication gaps and promoting inclusivity. The training process and fine-tuning of the model taught us the importance of extensive data and meticulous training to achieve accurate and reliable translations.

What's next for Speak2Sign

One of the exciting avenues for improving Speak2Sign is to expand its language capabilities by incorporating different languages within the Whisper AI model. As Whisper AI supports multiple languages, this enhancement will enable Speak2Sign to be used internationally, bridging communication gaps for people from diverse linguistic backgrounds. By incorporating multiple languages into the system, Speak2Sign will become a powerful tool for inclusive communication on a global scale. It will empower individuals who use sign language as their primary means of communication to interact more effectively with people who speak different languages.

Built With

• matplotlib
• pil
• python
• whisperai