Inspiration
Learning in linguistics about how sign language is specifically studied by linguists because of its evolutionary nature made us realize that many millions of signers out there may only be able to comfortably communicate with a small community of people. We want to fix this by building a mobile app that can translate a variety of international sign languages into written language.
What it does
We use a convolutional neural network to identify sign languages in a live video feed which translates signs in real time to English. Currently it can identify and translate from multiple sign languages into multiple written languages. We use a mobile app to use smartphone cameras to capture live video feed, using a take photo function to send images to the neural network to be processed and translated. We also save previous translations which can be viewed on the app.
How we built it
We built the backend of this project through the usage of OpenCV in order to seamlessly capture and display real-time video feeds. Moreover, we integrated CVZone, to detect intricate hand movements and predicts nuanced gestures. We used Expo to help build a mobile app on iOS using React-Native, Typescript, Javascript, Node.js, and Flask.
Challenges we ran into
We had a hard time transferring images that were taken from the phone on the app via the API to the machine-learning model as they were not technologies we were used to. We were successfully able to upload photos to the backend but was not able to return the translated language back to display on the application. There is also lot of similarity between different sign languages and even with thousands of sample sets, it is hard for computer vision alone to handle each language. In the future, we want to implement buttons like google translate to choose the language we want to translate for better accuracy.
Accomplishments that we're proud of
Although we were not familiar with developing mobile applications, we were able to complete a front-end IOS application with an interactive AI. We also were able to work with image files that were routed to the backend API after some struggles. Additionally, we were able to create our own machine-learning model using a data set that we personally built, leading to an accuracy rate of 99%+
What we learned
Learned how to implement and train a neural network, learned how to use React-Native, Expo, and flask. Learned about ASL and Korean Sign Language.
What's next for GST - Global Sign-language Translator
As our project relies on a machine learning model, amassing a comprehensive and expansive dataset becomes pivotal for the success and precision of our Global Sign-language Translator. Our vision encompasses the translation of an even wider array of languages and an expanded sign vocabulary. Additionally, we are committed to enhancing the user experience by developing a more feature-rich mobile application, ensuring that our solution is as accessible and beneficial as possible.
Log in or sign up for Devpost to join the conversation.