Inspiration
Which kid hasn't hoped for a synthetic instrument to be implanted deep in their brain so that they can hear different music based on what they are feeling? Well, we have good news for ya'll. You just need a camera in front of your face at all times and a powerful computer at your back. Or you can turn on your webcam while playing a video-game and let emoMuzik do its job to take your gaming experience to the next level.
Anyone who has played a game through and through is no stranger to how profound of an emotional experience a video game can be. And even more importantly, how profound the music score’s impact is on the gameplay experience. The combination of hearing the spine-chilling creepy music in the background as a door slowly closes on you during a horror game is what makes it an exotic and rare experience that brings the player back to the game multiple times.
What it does and how we built it
The code identifies the subject's face and subject's hand gestures, if any. It then sends the former to Microsoft Azure's Face API, which scores and returns the intensity of eight emotions: Anger, Content, Disgust, Fear, Happiness, Neutral, Sadness and Surprise. Based on these scores, the code decides the music that matches closest to the overall emotion of the subject and plays it. In parallel, the code uses Background Subtraction (to get grey color photo of hand), Contour Mapping, Convex Hull creation (recognition of hand) and Convexity Defects (recognition of fingers) to figure out the gesture. Based on the gesture, the next or previous song is played, or the music is paused and played. Finally, the code threads together the Facial recognition and Gesture recognition together.
Challenges we ran into
The project was an ambitious one to try for a 12 hour hackathon since we were dependent on an API that Google had spit out during the first five minutes of researching. Making the project work with a completely unknown API, learning several mathematical and imaging algorithms and methods from a GitHub user’s project and repurposing it to do the gesture control for playback of music were the major challenges that we throughly enjoyed climbing over.
Accomplishments that we are proud of
The ability to identify emotions of a human using the Face API from Microsoft Azure, playing relevant music to that emotion and also having gesture recognition work together, when each directive affects the other, was surprisingly hard to achieve and we are proud we could do it.
What’s next for Trial devpost
Our code currently requires frontal and near-frontal camera angles to detect emotions. More importantly, the API only supports a 1/3 frames per second rate for updating in the free tier, and can be improved up to 10 frames per seconds to give a faster response to changing emotions. emoMuzik would also benefit from a user friendly interface which allows the player to input custom playlists, add or remove songs on the fly and be easily be able to change the size of the gesture-space. A great twitch extension could be made for the streamers out there, allowing the viewers to add music to these playlists and make the gaming session more interactive. The gesture engine could also be improved to allow us to recognize more gestures, include volume control, scrubbing and playback speed.
Log in or sign up for Devpost to join the conversation.