Inspiration
We wanted to create an interface where you can make music with your body, similar to a theremin. Arms straight -> pure sine wave. Arms curved -> harmonics.
What We Learned
Computer Vision: We discovered that MediaPipe Pose can track 6 arm points in real-time, and that the path connecting shoulders -> elbows -> wrists can create musically interesting waveforms.
Audio Synthesis: We built a wavetable synthesizer from scratch and learned that different instruments need unique harmonic profiles. Adding the right overtones transforms a simple waveform into BASS, PIANO, STRINGS, or BELL sounds.
Real-Time Systems: Managing 4 concurrent threads (camera, serial, audio, sequencer) taught us about thread synchronization and the importance of proper debouncing – both in hardware and software.
UX Design: We learned that "preview mode" (hearing your shape while forming it) is critical. We also discovered that giving users musical default beat patterns means they can start playing immediately instead of programming from scratch.
Languages & Frameworks Python (NumPy, OpenCV, MediaPipe, SoundDevice, PySerial) C++ (Arduino firmware) Hardware Arduino Uno R3 Raspberry Pi 4B Webcam 12 push buttons + 5 potentiometers
Techniques Computer vision (MediaPipe Pose detection) Wavetable synthesis with harmonic profiles Real-time audio processing (44.1kHz) Multi-threaded concurrent systems Serial communication protocols
Log in or sign up for Devpost to join the conversation.