Inspiration
Have you ever seen a cool dance on TikTok? Do you know how to dance? For beginners, learning how to dance can be a daunting task. Even though there is a strong community with plenty of tutorials, it is not built for every single choreography in the world. Being able to learn one-on-one can significantly help newbie dancers to learn new choreo. Additionally, coming up with original choreography can be a struggle, as developing your internal database of moves to use can be challenging. Using your own eyes to judge your own clips can both feel embarrassing and inaccessible without a floor-length mirror. Taking inspiration from physical rhythm games such as Just Dance, Dance Dance Revolution, and osu!, we aim to expand on these functionalities to bring accessibility to beginner dancers who want to learn choreography at any time, bringing the lessons at home.
What it does
ReMix parses through uploaded videos, such as fancams and TikToks and automatically creates a beatmap using OpenCV. If uploading only audio, ReMix will generate an original custom beatmap using moves found in previously uploaded videos. After generating a beatmap, its your turn to learn the choreo through a learning-focused practice mode or test your knowledge through playing through the whole beatmap.
We are able to differentiate ourselves from our inspirations by emphasizing the learning aspect of these games, allowing dancers an environment to learn any choreography at any location. In our learning mode, ReMix will walk through a process similar to learning choreography in a dance studio. First, the user through all of the moves. Additionally, they would be able to run the choreography in a slower pace, allowing users to test their mastery of the choreography similar to learning choreography in person.
In our play mode, we run the choreography through the entire beat-map at 100% speed, giving the user a score based on how they match the choreography. Additionally, they will receive specific timestamps of any mistakes that are made within the choreography. This way, they are more objectively able to assess their mastery and go back to the learning mode to remaster the choreography.
How we built it
For our frontend, we utilized tools such as Claude Code and Google Antigravity to help develop a wireframe/prototype to help support our backend development. We later used shad/cn to bring a clean user interface to our users.
Challenges we ran into
Developing our model to score active video became a major challenge. We needed to do a lot of tweaking to fine-tune what is acceptable to earn points. We want to give enough freedom to express artistic changes, but ensure all major dance highlights are being met. This is solved by using MediaPipe to take account all the major key points on a person while dancing. We also ran into a merge conflict that led us to rebuilt some parts of this model also, especially with merging OpenCV pose models on top of both input video feeds.
We also ran into Git issues with our teammates pushing out of date code into our main overwriting our code but thankfully we were able to diagnosis the problem quickly and use our experience to solve it within 20 minutes.
Accomplishments that we're proud of
- Finish our MVP
- Our product generally works with most videos
What we learned
We learned a lot about better ways to utilize GitHub to allow for better communication. By creating pull requests, we were able to help resolve any merge conflicts and do code reviews. We learned that it is important to value and define Git etiquette, such as offering screenshots for front-end and providing descriptive commit messages to changes we make.
We also learned a lot about how to scope our hackathon projects and strengthened the need to focus on a MVP. We focused a lot on building a foundation to allow us to implement some of our stretch goals for ReMix also.
We learned how to use Google Cloud to get Gemini API keys and use their Cloud Storage to store our videos on their servers for their models to have quick access.
What's next for Remix
In the future, we would like to bring an UI overhaul to bring more personality to the application, to better mimic popular rhythm games such as osu! and Project Diva Mega Mix+. This way, we are able to bring the experience to be compatible with larger screens and bring a more game-like feel to the application. It would be nice to better optimize the live-feed to give real-time feedback to dancing and be able to send real-time API requests between the live video feed and the server. This way, users know how well they're doing throughout their dance.
Additionally, we would like to look into ways to take the user experience of ReMix outside of the standard mouse and keyboard by implementing motion controls utilizing OpenCV. This way, users are able to select their desired beatmap through their body, instead of running back and forth to their computer.
Lastly, maybe even a mobile app as well to allow users to take it on the go.
Sources
Humanoid 3D Model: https://sketchfab.com/3d-models/low-poly-stick-figure-rigged-47e49c8f24d14fb7a3d477640e3d0cf2
Log in or sign up for Devpost to join the conversation.