SignSync

Flow chart explaining the work-flow
Hardware used for the project
UI - Hearing individual's point of view ; Top Section - detects/ takes in input of ASL | Bottom Section - outputs written translation of ASL
UI - Hard of Hearing/ Deaf individual's point of view; Written translation of Hearing individual's speech

Inspiration

The inspiration for this project is derived from the research presented in https://www.ijspr.com/citations/BSIP2018/IJSPR_BSIP_201814.pdf

What it does

SignSync is an American Sign Language Recognition tool designed with CNN Model Architecture for use in commercial and service areas to improve accessibility and facilitate seamless communication between ASL-speaking customers and non-ASL-speaking staff. Our goal is to enhance our model to interpret various words and sentences in ASL and subsequently translate them into text for individuals who do not speak ASL. This product will be available to staff working in retail or service spaces. When an ASL-speaking customer wants to place an order, the device can be set up within the rotatable tipping screen or any other pre-installed device. It will capture the customer's hand gestures and movements, translating them into English text that the staff can read to fulfill the customer's request.

How we built it

For our project, we are utilizing a Convolutional Neural Network (CNN) to recognize American Sign Language (ASL) gestures. The Python script listens for input from the ultrasonic sensor on the breadboard, which detects deliberate movement. Then, when deliberate input is detected (when someone's hands come near the sensor) the Arduino code is executed. This triggers the webcam to open on the laptop and start recording the visual input. The process ends when the user clicks submit (aka, hitting q on the keyboard) and this recording is saved, ready to be processed by the machine learning model. Our machine learning model captures visual input and applies the openCV library to detect and classify ASL hand gestures. we trained our model on an open source ASL Alphabet Dataset(https://www.kaggle.com/datasets/grassknoted/asl-alphabet).

Challenges we ran into

We encountered significant challenges while trying to integrate OpenCV into our project, particularly during the installation phase. Despite multiple attempts, the process repeatedly got stuck during the build stage and never fully completed. This persistent issue prevented us from successfully downloading and utilizing the cv2 module, which was crucial for our machine learning tasks involving computer vision. As a result, we were unable to proceed with using OpenCV for image processing, which significantly delayed our progress.

Accomplishments that we're proud of

Despite the difficulties we faced with the machine learning aspect due to issues with OpenCV, we are proud of the progress we made on the hardware side of the project. Our Arduino setup, which includes the ultrasonic sensor and webcam integration, functions as intended, effectively detecting hand movements and triggering the recording process. This solid foundation in hardware gives us confidence in the potential of SignSync. Additionally, we successfully trained our model using the ASL Alphabet Dataset, demonstrating our ability to work with machine learning models even though we couldn’t fully implement it due to the technical challenges with OpenCV.

What we learned

Through this project, we learned a great deal about the importance of integrating each component in a system and how technical difficulties in one area, such as the failed installation of OpenCV, can impact the overall functionality. This experience highlighted the necessity of contingency planning and the ability to adapt when things don't go as expected. We also gained practical experience with Arduino and hardware integration, which was a valuable learning opportunity. Finally, we deepened our understanding of machine learning, even though we couldn’t fully implement the model, and recognized the challenges that come with integrating software libraries and dependencies in a complex project.

What's next for SignSync

This project holds significant value for us, and we aim to continue development beyond the hackathon due to its potential impact on society. We plan on implementing a two-way communication form in the future. Not only will the ASL be translated to text, but the staff using this tool will be able to type/speak into their device which will be presented to the customer as text to increase accessibility and foster inclusivity. Another vision we have for SignSync is to improve the image detection further making it easier for the machine to pick up on gestures with better accuracy. For further insights into the statistics of ASL speakers in the U.S., refer to this resource(https://www.kaggle.com/datasets/grassknoted/asl-alphabet).