Inspiration
Always loved melodic music, and have been watching the youtube videos of people playing piano with the Synthesia game, and I had always wanted to try it out; The only problem I don't have a piano and I don't want to spent my hard earned money over it; Solution: create your own piano playing experience from a sheet of paper and make a Synthesia inspired game to go along with it.
What it does
Keyzz transforms your laptop into an interactive, gamified piano, allowing you to enjoy playing music without needing an actual piano or additional hardware. Leveraging computer vision, machine learning, and real-time audio feedback, Keyzz provides an immersive piano-playing experience right from your home.
How we built it
Vision-Based Key Detection: Detects and maps printed piano keys using just your laptop's webcam (with the help of openCV and MediaPipe). Real-Time Audio Playback: Plays notes instantly based on finger placement and movement (with pygame) Interactive Game Mode: Engage with falling notes similar to rhythm-based games, enhancing your musical learning experience (with pygame). Instant Calibration: Quickly stabilizes and calibrates detection for consistent and reliable performance.
Challenges we ran into
- A issue that took me more than half of my time hacking was that the application was registering that the user had clicked a piano key, but somehow the application would not play the correct corresponding notes for the pressed key (the answer: bottleneck in the audio channels for pygames)
- Being limited to only a single camera raises a lot of problems, one of them being the captured information being limited to 2 dimensions only, this causes multiple issues for working with multiple fingers. (solution: use a pre-trained CNN model for depth perception through a single camera image, MiDaS; this solution is there on the model branch of GitHub; it requires a powerful GPU, which my machine locally could not sustain.)
Accomplishments that we're proud of
You can actually pay a rhythm game through a printed sheet of paper and get a glimpse into what it is to play a real Piano. Anyone around the world can experience by just following 3 simple steps mentioned on GitHub README.md file. So go ahead and compete with your friends to see who can get the highest score in this new experience of piano rhythm game.
What we learned
How to do hand-pose estimation with openCV and media-pipe; and how much liner-algebra one needs to know to distort a simple mapping to the actual live feed from the camera.
What's next for Keyzz
- Make it modular, right now it only supports one major Octave, but I want to make it such that, if another sheet of paper is printed, one could place them side by side and the app would recognize the addition of another octave and suggest songs and lessons accordingly.
- Add alot more songs; being only limited to one Octave really narrowed down the potential songs I could choose, so would definitely like to expand upon the whole song selection. (also all songs need to be hard-coded by hand, making it a very time consuming task to perform)
- Add more gamification elements to it, and improve upon the user interface.
Log in or sign up for Devpost to join the conversation.