SignStream

SignStream creates a virtual ASL interpreter that translates speech from online videos in real-time.

Comment

Inspiration

Currently, there are over 360 million people across the globe who suffer from disabling hearing loss. When these individuals want to watch videos or movies online, they have no choice but to read the live captions. Studies have shown that individuals who are hearing impaired generally prefer having a sign-language interpreter rather than reading closed captioning or reading subtitles. However, ASL interpreting technology available to the public is minimal. This is what inspired the creation of SignStream. We identified a need for this technology and built what we believe is an innovative technological solution.

What it does

SignStream creates a *virtual ASL interpreter * that translates speech from online videos in real-time. Users play videos or audio and an ASL translation is returned.

How we built it

The initial design of the website layout was made on Figma. Unique graphics to SignStream were created on Photoshop and Adobe-Illustrator.

The frontend components including references to pages and interactive features were built through React.js.

To build the backend, Python and Beautiful Soup were first used to web scrape and store jpg/mp4 files of ASL vocabulary.

The AssemblyAI API was used to create the portion of the program that receives audio input and returns text.

Finally, the backend was written in JavaScript and interpreted by the website using Node.js.

Challenges we ran into

The main challenge we faced was displaying the jpg/mp4 files onto our website in real time as audio was being input. This was primarily due to the fact that not all words can be displayed in ASL. This delay was created due to the run-time of sorting through the text output to identify words. However, by taking time to improve our code, we were able to reduce the delay.

Accomplishments that we're proud of

As a team of first year students, we had very limited prior knowledge of web development. Elements such as coding the backend of our website and figuring out how to use the AssemblyAI API within 36 hours proved to be quite the challenge. However, after hard work and collaboration we were able to successfully implement the back end and feature a real-time streaming page.

What we learned

During our QHacks weekend, every team member gained precious knowledge about web development and problem solving.

Our frontend developers increased their ability to build a website using React.js. For some of us, this was our first time working with the library. However, by the end of the weekend we created a fully-functioning, responsive website.

The backend developers learned a lot about working with APIs and Node.js. We learned the importance of persevering when faced with challenges and collaborating with fellow developers.

What's next for SignStream

In the future we are looking to expand SignStream to provide an optimal all around user experience. There are a few steps we think could benefit the user experience of SignStream. First, we would like to expand our database of ASL vocabulary. There are over 10 000 signs in the ASL language and expanding our database would allow for clearer interpretation of audio for users.

We are also looking to implement SignStream as a Chrome/Web extension. This extension would have the virtual interpreter appear as a smaller window, and reduce the amount of the display required to use the program.

Built With

Share this project:

Updates