MusicalTouch

Inspiration

Creating music feels locked behind a large wall of talent that most people find hard to overcome. Inspired by the idea that real-time hand gestures could shape and guide sound as intuitively as a conductor leading an orchestra, unlocking a more natural and immediate bond between human movement and musical production.

What it does

It is a hand gesture based music composer that lets you use hand motion to switch between different instruments, activate their sounds, adjust chord progressions or patterns, adjust volume, tempo as well as making music composition feel more hands-on and intuitive.

How we built it

Utilizing Haar Cascading through Mediapipe for real-time hand detection and tracking as well as OpenCV to determine gestures through capturing video frames. Created an on-camera interface that translates gestures into UI navigation. Utilizes Mido and FluidSynth to synthesize the chords and run them in real-time rather than have pre-recorded audio files.

Challenges we ran into

Differentiating the different gestures from each other, determining the appropriate libraries to integrate the many components, and deciding whether to use pre-existing music or compose original chords.

Accomplishments that we're proud of

This project involved extensive research and the learning how to utilize the various libraries that had not been previously used. Considering the time all of that took, the successful development of a fully functional application that closely aligned with the original vision is something we are proud of.

What we learned

One take away from this experience is that there is always something more to learn and no matter how inexperienced one might think they are with certain things, one is more than capable of teaching him or herself what is required and succeed even in a short amount of time.

What's next for MusicalTouch

In the future, MusicalTouch has the potential to be further developed with additional tracks, more complex composition styles, and advanced gesture recognition capabilities. By incorporating a wider range of musical genres and enhancing the user interface, the application could offer a more immersive and dynamic experience.