Dance Heroes

Title Screen
Music Upload
Model Selection

Inspiration

Colin sat with his usual yogurt cup in hand, watching his brother play Just Dance. Longing to play, he grabbed the controller and scoured the song list for his favorite artists: Stray Kids, Seventeen, NIKI. But with each passing title, his heart sank—none of them were there. Frustrated but determined, Colin vowed to change that.

From that moment, Dance Heroes was born—a vision to bring everyone's favorite songs to life on the dance floor.

What it does

Simply drag a .WAV file into Dance Heroes, and our game will generate AI-driven choreography tailored to your chosen song (or sound). You'll follow a dancing model as you perform the routine to earn points. Once you’ve completed your dance, you can view your final score and see how well you performed.

How we built it

Using MediaPipe, we capture the user's body landmarks and map their movements onto a 3-D model inspired by SysMocap. At the same time, we generate choreography for a chosen song using EDGE: Editable Dance Generation From Music and create a Blender 3-D model that performs the routine. This generated model is then mapped onto the SysMocap-inspired framework. By comparing the positions and coordinates of the user and generated models, we calculate a score based on the accuracy of the user's movements.

Challenges we ran into

We ran into early problems when trying to map both the generated model's and user's coordinates onto a 3-D model figure. Luckily, we were able to resolve this problem utilizing Blender as well as taking inspiration from SysMocap: a software used to capture real-time face/body motion to render onto a 3-D model for VTubers, VR, AR, and Live.

Accomplishments that we're proud of

Mapping the generated AI choreography onto a 3-D model has been, without a doubt, the most exciting achievement of this project. We're also thrilled with how accurately the user's coordinates were mapped onto the model, as well as the functionality and accuracy of our point system.

What we learned

We learned a lot about Computer Vision and Audio Process with the use of libraries and tools such as MediaPipe, SMPL datasets, THREE.js. In addition by implementing a live 3-D model that tracking real time player's movement we learnt about the use of Blender and VRM and FBX files.

What's next for Dance Heroes

The next step is to implement a leaderboard, allowing users to compare their scores with friends or track their progress against their previous performances. We also plan to introduce a difficulty setting, enabling players to adjust the complexity of the choreography. Finally, we aim to add a multiplayer mode, allowing users to play with or compete against their friends in real time.