Baseline look
Logo

Inspiration 💭

Traditional piano education requires expensive instruments and lessons, creating barriers for many aspiring musicians. We wanted to democratize music education by creating a virtual piano studio that works with just two webcams.

What it does 👩‍💻

AiRPiano transforms any space into a virtual piano studio using computer vision. It tracks your hand movements through a webcam to play a virtual 3D piano, provides guided lessons with real-time feedback, displays sheet music, and lets you learn songs with falling note visuals.

How we built it 📦

We used Three.js for 3D graphics and a forked repository for initial movements and models then augmented them in Blender, MediaPipe/TensorFlow.js for hand tracking, MIDI.js for audio, and OpenSheetMusicDisplay for sheet music. The platform runs entirely in the browser using WebGL, Web Audio API, and computer vision.

Challenges we ran into

Integrating multiple complex technologies (3D graphics, computer vision, audio synthesis) Achieving accurate hand tracking and pose estimation that translates to precise piano playing Synchronizing the animated pianist with real-time hand movements Creating a calibration system that works for different users and environments

Accomplishments that we're proud of 🎊

Creating a fully functional air piano that requires no physical instrument Building a real-time feedback system that validates note accuracy Developing an immersive 3D experience with an animated pianist Making music education accessible to anyone with a webcam

What we learned

How to integrate computer vision with 3D graphics and audio in real-time The complexities of hand pose estimation and mapping to musical interfaces The importance of calibration systems for personalized user experiences How to make advanced technology accessible and user-friendly

What's next for AiRPiano

Expanding lesson library with more exercises and songs Adding multiplayer features for remote lessons Supporting more instruments beyond piano Mobile app development for greater accessibility Advanced progress tracking and personalized learning paths