Airwaves

What it does

Airwaves is a rhythm-based browser game that uses your webcam to detect hand movements and translate them into mouse clicks, scrolls, and gestures. The player hits on-screen targets to the beat—entirely hands-free. Behind the scenes, we interpret specific hand poses and motions in real time, making the game both a fun challenge and a light physical therapy exercise to improve hand mobility, coordination, and range of motion. The game also supports Clone Hero mapping files, which we plan to expand with a custom upload feature for players to add their own songs.

How we built it

We used TensorFlow.js and the HandPose model to detect and track hand landmarks from a webcam feed, powering all in-game gesture recognition and allowing the experience to be fully web-based. OpenCV handled all non-gameplay interactions, including menu navigation and volume control through hand gestures. The front end was built with React and Three.js, using React Three Fiber for rendering 3D elements, with custom CSS animations for a smooth, immersive rhythm experience. The hand motions were designed to replicate common hand mobility exercises used in physical therapy , making gameplay beneficial for hand mobility and coordination.

Challenges we ran into

Achieving consistent hand detection across different lighting conditions
Filtering out accidental movements while keeping input responsive
Calibrating gestures to ensure therapeutic value without causing fatigue
Balancing accessibility and rhythm mechanics without one compromising the other
Integrating real-time hand tracking with the rhythm-based gameplay mechanics

Accomplishments that we're proud of

Developed a full working prototype of a webcam-controlled rhythm game in under 36 hours
Created a gesture-to-input system that could have real therapeutic benefits
Designed an experience that’s fun for everyone, but especially impactful for users with limited mobility or in recovery
Successfully integrated real-time hand tracking with a 3D-rendered game environment, fully rendered in React

What we learned

How to design interactive systems that are fun and functional
That games can be more than games—they can be tools for well-being
The importance of balancing technical complexity with user experience

What's next for Airwaves

Develop new levels with progressively challenging therapeutic motions
Add hand tracking analytics to help users (and therapists) monitor progress
Integrate accessibility settings like motion sensitivity, tempo adjustment, and guided movement training
Optimize the hand tracking pipeline for better performance on low-end devices