SignScript

Empower Your Voice with ASL

Comment

Inspiration

The inspiration for SignScript came from recognizing the communication barriers faced by the Deaf and hard-of-hearing community, making up 20% of the world population (1.5 billion people). We wanted to create a tool that not only makes learning American Sign Language accessible and engaging but also fosters understanding and connection between the hearing and Deaf communities. By leveraging Machine Learning and Software Engineering, we aim to bridge these gaps and promote inclusivity, bringing the voice of this community to the general people.

What it does

SignScript provides a typing game where users can customize the time limit and word count. Instead of practicing keyboard typing, users practice signing the ASL alphabets. The game is designed to be engaging and educational, helping users to improve their signing skills in a fun and interactive way.

How we built it

SignScript's interface and game logic is built using React.js as a web application. So, it can also be use on a multitude of devices that have access to browsing. The detection model is a built-in model from TensorFlow.js, which gives us the hand landmarks (key points) for drawing the hand figure and use as feature for our prediction model in real-time. The ASL prediction model is train on this Kaggle dataset on Colab using Tensorflow, and Scikit-Learn with a 99.4% accuracy, which is then exported as a JSON to be loaded onto the web app. Since the bulk of computation is done on the user's machine through TensorFlow.js, the application does not cost anything for upkeeping, and can be deployed anywhere.

Challenges we ran into

We had a rough time coming up with an idea considering the relatively vague prompt. At first, gesture recognition was a tremendous challenge, we tried different approaches from comparing it to a predetermined set of landmarks for each ASL hand sign to using a self-calibrating method for each letter, which took a large amount of time. However, those come with limitations, either a lack of diverse camera perspective or a rigid comparison algorithm. That's where we decided to use a classifying model that is train on a large dataset, which yields the highest performance.

Accomplishments that we're proud of

We are extremely proud of the result of the project, being a vague idea to a successful implementation. Moreover, the relentless effort of every team member during the making of this project is already a strong win for us.

What we learned

Coming in with little to no experience for the different technologies of the project, like TensorFlow, Machine Learning, and more, we were still able to implement a moderately complex system and develop a more technical understanding. Moreover, we also learn the importance of designing with the end-user in mind, understanding how to create solutions to the people's problems

What's next for SignScript

There are still many minor problems with the current state of the application such as recognition capability for some letters and the performance issues for some devices that lack computation capability. Therefore, we look to first address and remove these issues for the best possible experience. In addition, the application also shows many potential for more exciting features like a multiplayer mode for sign racing, a statistics to keep track of personal metrics, and forums or medias to connect people in the community.

Built With

Share this project:

Updates