Inspiration
Learning a TikTok or K-pop dance can be awfully difficult and is quite manual and confusing. You have to pause the video, rewind it, pause it again, then rewind for every single step you are trying to learn. Sometimes tutorials lack verbal or textural steps as well to guide you on how you should move. I personally struggled when someone asked me to learn a dance and I took ages attempting or even trying to learn a dance I think would be fun but would end up frustrated. I wanted to build a tool to address all of these issues with many opportunities to continuously improve.
What it does
choreo turns any dance video (mp4, MOV, or even a TikTok link) into an interactive step by step tutorial. When you upload a file or a TikTok link, the pipeline will generate a skeleton overlay using computer vision, then use Google Gemini's vision AI to automatically extract every distinct move with timestamps, titles, and beginner-friendly descriptions. On the results page, steps are both displayed and clickable. In the controls below, you can loop a step, slow down the video, mirror it, and practice alongside your webcam.
How I built it
Frontend: React with React Router and CSS components Backend: Python + Flask REST API Computer Vision: Google MediaPipe for skeleton mappings AI Analysis: Google Gemini 2.5 Flash model API to analyze video and extract steps Database: SQLite from Flask-SQLAlchemy for saving tutorials to a personal library Auth: Flask-Login and Flask-Bcrypt for secure user accounts
Challenges we ran into
I ran into dependency issues when trying to start the backend since I was integrating both Gemini and MediaPipe. I also ran into some issues with video syncing and steps getting messed up due to hallucinations from Gemini providing incorrect data which as a result ruined the user experience for those trying to learn, but I was able to overcome these challenges and continue further.
Accomplishments that I'm proud of
I do not have as much experience with full stack web development and it took me some preparations leading up to the hackathon to test the skills that I learned before it started. I did not think that I was able to get as much done and was worried that I would end up with something very empty. I
What I learned
Building a full stack application with a whole pipeline from scratch was new for me since I've only worked with really basic projects leading up to the hackathon. I was able to get a deeper understanding with React and also was able to grasp a better understanding of making requests to the backend and how we can use the response that is sent back to the client. There were some issues that I also ran into with the libraries that simply took reading documentation to fix the problem.
What's next for choreo.
I have a lot planned for the application especially since there was a time constraint I had to follow. Since people can create their own accounts to store their tutorials, I was considering adding a feed where you can upload your saved tutorials for people to see. Other users have the ability to view this and can 'like' the tutorial to send it higher in the feed to make it more popular. I believe this benefits everyone as it helps the artists who created the songs, the video creators, and users who want to learn a new dance. I also plan on integrating MediaPipe with the live webcam with the capability to record and get an accuracy score by comparing the source's skeleton with your own. I believe there is a lot to add and I am open to taking any suggestions on the way!
Log in or sign up for Devpost to join the conversation.