ViSign

Empowering inclusivity through captioned signs.

Comment

Inspiration

One of the greatest attributes of an engineer is their ability to solve problems, but it is the opportunity to use these skills to help those around them that truly fills their heart with satisfaction and joy. As a group of three engineers and friends from India, we cherish our daily commutes to and from college, discussing coursework and sharing technology news. But on one such ordinary day, fate introduced us to a small deaf boy who touched our hearts with his enthusiastic and determined attitude to communicate with us through writing. Each word, each smile, and each facial expression was a testament to his unwavering strength and resilience. Witnessing his unwavering efforts to feel included and connect with us was a humbling experience, and it left an indelible impression on our hearts. We were inspired to do something for our fellow physically challenged brothers and sisters, and after a day of brainstorming, we came up with ViSign. This software is our way of extending a helping hand to the deaf and mute community, ensuring that they too can be a part of our world without any barriers.

What it does

ViSign is a web-based application designed to overcome language barriers for individuals who are unable to hear or speak. It serves as an online platform where disabled individuals can create meeting rooms to communicate with each other using sign language. Our software captures these signs and converts them into alphabets, allowing for seamless communication and understanding between individuals. We have also incorporated an additional chat functionality in our application that allows for further seamless communication.

How we built it

Our team utilized the power of Machine Learning to create a robust model capable of accurately classifying signs and converting them into alphabets. To achieve this, we first collected and curated a vast amount of necessary data. Using the data collected, we trained our Machine Learning model using Teachable Machine. Once we had our model in place, we created a user-friendly meet application using webrtc, express, and socket.io to facilitate seamless communication between individuals. We also developed a server in Python that accepts a data stream from the meet application and generates predictions that are displayed in caption format. Bringing everything together, we carefully integrated all the different components while making modifications and fine-tuning them until we achieved the desired output for the signs provided.

Challenges we ran into

  1. Finding adequate data samples in diverse formats to input and train our model was one of the main difficulties we encountered. We diligently searched through a variety of sources to compile a significant quantity of different data that would allow our model to understand and correctly categorize a variety of sign language gestures. The data samples were carefully chosen and curated to be of the highest quality and sufficiently diverse to successfully train our model. This procedure was crucial to the success of our application since the caliber and variety of the data samples we utilized directly affected how accurate our model would be.

  2. Our team encountered various challenges during the data processing phase, particularly with signs that appeared similar to each other or had minimal differences. For example, distinguishing between "6" and "W," or "H" and "U" required meticulous attention to detail and an extensive amount of data processing. We meticulously processed the data to ensure that the model accurately distinguished between these similar signs, enabling seamless communication and accurate translations for individuals using our application.

Accomplishments that we're proud of

We are extremely proud of the numerous accomplishments we achieved during the development of our ViSign project.

  1. Stepping out of our comfort zone to participate in the hackathon, gaining new skills and knowledge
    1. Out-of-the-box thinking to create a unique solution that bridges the communication gap for hearing and speech-impaired
  2. Overcoming various hurdles in collecting data samples and building a machine-learning model Creation of a user-friendly web-based program for sign language communication, eradicating linguistic barriers and fostering inclusion
  3. Effective use of machine learning to interpret sign language gestures into readable alphabets for seamless communication for the deaf
  4. Strong data processing pipeline to choose high-quality data samples, essential for increasing machine learning model accuracy
  5. Potential to improve the quality of life for thousands of people with disabilities, enabling them to communicate with ease and comfort. Our prototype has the potential to improve the quality of life for thousands of people with disabilities by enabling them to communicate with ease and comfort, and we put in place a strong data processing pipeline that was essential for training our machine-learning model and increasing its accuracy.

What we learned

Throughout our journey of building a web application that enables communication for the hearing and speech-impaired through sign language recognition and translation, we gained a wealth of knowledge and experience. In this section, we would like to reflect on what we learned from this project, both in terms of technical skills and personal growth.

  1. New technical skills such as Machine Learning, WebRTC, and data processing pipeline creation.
  2. Problem-solving skills and critical thinking through tackling challenges during the project.
  3. The importance of teamwork and collaboration to achieve common goals.
  4. The satisfaction of contributing to a project that has the potential to improve the lives of people with disabilities.
  5. The experience of participating in a hackathon and the benefits of engaging in extra-curricular activities.
  6. The value of perseverance and determination in completing a challenging project within the given timeline.

What's next for ViSign

Certainly! Our project's long-term goal is to keep enhancing and extending its capabilities in order to better the communication experience for people with impairments.

To give more complete communication possibilities, one approach we may take is to incorporate additional sign language movements into our model. We may also look at including tools like text-to-speech and voice recognition to make it easier for those who have trouble signing to communicate.

We might also try to reach more people with our web application by making it more usable in areas with poor bandwidth or restricted internet connection. This might entail designing an offline version that can be downloaded and used without an internet connection or optimizing the program for mobile devices.

Share this project:

Updates