Serial Signer

✨ Inspiration

We embarked on this project with a motivation: reduce the communication barriers faced by individuals with hearing impairments. There are over 70 Million deaf people around the world who use sign-language to communicate (src). That is a relatively big percentage of people, whom we can encounter in our daily lives. While there are many projects that aim to to facilitate understanding of sign language (through arm-movement translations), it's important to recognize that the majority of people in the world don't have any knowledge of sign language. As a result, many individuals lack the ability to communicate effectively with those who have hearing impairments. Our inspiration was to create a solution that enables seamless communication with individuals who have hearing impairments, regardless of prior knowledge of sign language.

🌎 What it does

Serial Signer is an AI application which translates spoken or written language into sign-language, a method of communication used by the majority of people with hearing impairments. Users are able to speak or write text, which Serial Signer translates into easy-to-understand animated hand signals.

⚙️ How we built it

diagram We build our frontend using basic HTML, CSS and JavaScript. In order to render the 3d sign language animations, we used Three.js. We parsed the American Sign Language Dictionairy (src), for videos of the most common words being signed. Using MediaPipe (Google Tech), OpenCV, and Numpy on top of Google Colab, we were able to obtain coordinates for each joint of the hand, during each frame -- which were then rendered as a series of cylinders and spheres which were attached to one another in order to represent each bone segment and joints of the hand.

🚧 Challenges we ran into

We have not previously used Three.js so figuring out how to render and animate 3d objects in the browser proved to be a steep learning curve and a difficult challenge. Another issue we ran into was how difficult it is to leverage AI into the project based on the limited amount of computing power which we had access to. This was overcome by time managing properly and starting all the complicated computing first while simultanoulsy coding the app itself. On top of that, none of us have worked with this type of recognition AI and librarires before, hence the training process took a long time.

😁 Accomplishments that we're proud of

We are proud of overcoming the many challenges we faced while building the project as mentioned above. We are proud of successfully creating an application that allows users to translate their voice into Sign Language. Narrowing the gap of communication with people who have hearing impairments. We are proud of completing a project working in a team of only two people, which none of us have done before.

📚 What we learned

We learned many new technologies such as:

Three.js: we learned how to use Three.Js in order to render and animate Three Dimensional objects in the browser
- MediaPipe: we learned how to use Google MediaPipe in order to track hand location
- Google Colab: we learned how to trains our models and use google cloud resources

🤔 What's next for Serial Signer

We have many ideas and plans for Serial Singer. We envision it as a tool that will eventually completely eliminate the border between spoken language and sign language. We hope to eventually add :