Inspiration
- Our inspiration comes from the movie CODA, which stands for Child of Deaf Adults. This movie increased our awareness of people with disabilities and thus made us ponder ways to help them. We try to find ways to make society more inclusive of deaf people, which is a minority group. We think it would be beneficial to increase the spread of American Sign Language (ASL). If more people know ASL, more people could communicate with deaf people and thus make them more integrated into society.
- We need an incentive for the public to learn ASL, and we think the best approach is to gamify the learning experience. We think that a music rhythm game is perfect for the job. Rhythm games typically require players to hit buttons in a specific order as seen on the screen, while following the beat of the music, which is fun for the players as they get to rock out to their favorite tunes, thus making rhythm games very popular. We see this as an opportunity to use the player’s enthusiasm toward learning ASL! Thus, we decided to add a twist to the rhythm games: instead of having players hit buttons, they will be waving sign ASL signs to the beat of the music!
What it does
CODA is a music rhythm game that teaches players American Sign Language (ASL) while having fun! Players will wave ASL signs to the lyrics of a song in front of a camera. Our game scores the players based on the accuracy of their waved hand signs as well as their adherence to the beat of the song.
How we built it
- Vision: the tracking model was based on a trained dataset from google’s mediapipe, which was trained on hands source from a variety of contexts. This allowed the dataset to recognize any pair of hands we tested on it. The data processing was completed using the opencv and numpy python libraries.
- Game: The user-facing aspect of our project was developed in Unity game engine. It offered a 2D game creation environment that allowed us to focus on creating a game that met our requirements and aesthetics.
- Graphics: The ASL signs were hand-drawn in Procreate from reference photos and then imported into Adobe Photoshop where they were colored, textured, and all-in-all “retrofied.” Elements were also animated in Photoshop through stop-motion frame rendering. Pixel icons were created in Piskel. The dynamic backgrounds of our songs were generated with Veed.io.
- UI/UX: We used figma to design our UI, which is retro desktop themed that matches the theme of HackGT 9. We also created a website where users could learn more about our game. In the future, we will include a link where our game can be directly downloaded from this website
Challenges we ran into
- Lack of palm images dataset to train our machine learning algorithm to detect the hand signals.
- ASL signs had to be limited to stationary ones only, as the ability for our machine learning algorithm to differentiate between moving gestures was limited.
- Signs and lyrics had to be synchronized manually, which was a time-consuming and tedious process.
- It is hard to distinguish between certain hand signs, especially for a camera that doesn’t have any true depth perception
Accomplishments that we're proud of
- We overcame the limited dataset challenge and developed an algorithm to accurately detect hand signals!
- We made nearly all of our game assets (logos/images/icons/UI elements) from scratch through a combination of hand-drawn images, vectors, and animations.
- Smooth, engaging gameplay with an attractive retro aesthetic.
What we learned
- How to do UI/UX design with Figma
- How games and machine learning algorithms are made
- How to animate in Photoshop and Procreate
- Photoshop techniques for graphics design
What's next for CODA
- Add more complicated songs to our song bank so that the user can learn more words in ASL.
- Improve the accuracy of our machine learning algorithm.
- Refine the machine learning algorithm to account for kinetic ASL words and phrases.
- Develop scripts to automatically calibrate songs (matching lyrics to the corresponding ASL signal) so that players can add any song that they want (instead of having to choose from our song bank). This will increase the popularity of our game as players can play their favorite music.
Log in or sign up for Devpost to join the conversation.