VibeVirtuoso is a gesture-controlled music system designed to empower people with mobility impairments and support music therapy in new, accessible ways. By turning simple hand gestures into rich, expressive music, VibeVirtuoso allows anyone — regardless of physical ability or musical training — to become a performer, a composer, and a creator. Whether it's waving a hand to play a violin or flicking fingers to build synth layers, VibeVirtuoso brings a sense of play, creativity, and agency to users who may not otherwise have access to traditional instruments.
Why We Built It:
We were inspired by the powerful role music plays in healing, expression, and connection — especially for those who face physical barriers. We imagined a system that could be used:
By individuals in music therapy sessions
In rehabilitation environments to encourage motor engagement
As an expressive tool for people with limited mobility or dexterity
In special education classrooms or therapeutic arts programs
We wanted music to be accessible to everyone, no matter their physical abilities
How We Built It:
We used OpenCV and MediaPipe Hands to detect hand gestures in real time, enabling intuitive interaction with virtual instruments. Gestures were mapped to control various musical elements such as notes, chords, pitch, vibrato, and volume. For high-quality sound playback, we integrated FluidSynth along with the FluidR3_GM.sf2 SoundFont, allowing users to play instruments like the violin, saxophone, flute, drums, guitar, and piano synth. To support data analysis and potential future AI training, we utilized MongoDB to log gesture events and musical sessions throughout user interactions.
What We Learned:
We learned how to design music systems for non-musicians and individuals with varied motor abilities. We also recognized the emotional and therapeutic impact of accessible music creation. A key technical focus was minimizing gesture-to-sound latency, and we used MongoDB for structured music data logging to support analysis and future development. The challenges we faced included creating gesture mappings that worked across a wide range of motor skills and achieving smooth, real-time tracking using webcam input. We also worked on making synthetic instruments sound expressive and human. Additionally, calibrating gestures to be both intuitive and accessible was a critical and ongoing task.
What’s Next:
Future improvements for VibeVirtuoso include adding a visual user interface that allows therapists to customize instrument mappings based on individual user needs. We also plan to enable recording and playback functionality to support musical journaling and progress tracking. Additionally, we aim to integrate AI that can suggest emotions or musical ideas based on gesture dynamics, enhancing both expressiveness and therapeutic value. Finally, we intend to deploy the system in schools, therapy centers, and hospitals to make accessible music creation available to a wider community.
TL;DR:
Music should be for everyone. VibeVirtuoso lets users create, express, and heal — using only their hands. It’s not just a tool — it’s a stage, a studio, and a voice for those who need it most.
Log in or sign up for Devpost to join the conversation.