StepSync

Your personal Dance Tutor dedicated to making you a rockstar on the dance floor

Comment

About the Project

Inspiration

The concept for this real-time dance learning and sharing platform emerged from our collective passion for dance and technology. We envisioned a platform that not only democratizes access to high-quality dance education but also fosters creativity and originality. Inspired by the challenges of learning from static tutorials, we aimed to create an interactive system that provides real-time feedback, enables users to share their creations, and ensures proper attribution for original dance moves. Our goal was to merge cutting-edge technology with a user-friendly design to revolutionize how people learn, share, and innovate in the dance world.

Features of the Platform

Our platform is designed with three core features that work together to create a comprehensive ecosystem for dancers: social sharing, learning feedback, and originality tracking. Here’s a breakdown of how each feature works:

1. Social Sharing

  • Upload Your Dance Videos:
    Users can upload their own dance videos to the platform, showcasing their creativity and allowing others to learn from them.
  • Learn from Others:
    Users can browse through a library of dances shared by others, selecting routines they want to learn or draw inspiration from.
  • Community Engagement:
    The platform functions like a social media space for dancers, where users can interact with each other’s content by liking, commenting, or sharing.
  • Discover New Talent:
    The platform encourages collaboration and discovery by highlighting trending dances or showcasing creators who consistently upload unique content.

2. Advanced Learning Feedback

  • Real-Time Pose Comparison:
    Users can upload a reference video (either their own or one from the platform) to compare their movements in real time using MediaPipe’s pose detection.
  • Node-Level Analysis:
    The system identifies specific body parts (nodes) that are misaligned during each frame of the routine.
  • Timestamped Feedback:
    Deviations are mapped to precise timestamps in the video, so users know exactly when and where they went off track.
  • Actionable Corrections:
    The platform doesn’t just highlight errors—it explains how to fix them by providing detailed guidance on adjusting angles or positions of specific nodes.
  • Progress Tracking:
    Users can track their improvement over time as they practice routines, receiving updated feedback on areas where they’ve improved or still need work.

3. Originality Tracking

  • Patent Your Dance Moves:
    When users upload a new dance, it generates a JSON file representing detailed pose data alongside the video file. This creates a digital signature for the routine.
  • Pinata Vector Database Integration:
    Using Pinata’s vectorized database, the system clusters similar dances based on metadata derived from JSON pose data.
  • Overcoming Embedding Challenges:
    Initially, we struggled with creating robust embeddings for originality tracking. Traditional embedding techniques were inconsistent. However, we resolved this by directly using JSON pose data as vector representations, which provided clear and comparable results.
  • Originality Verification:
    When a new dance is uploaded, the system compares its vector representation against existing records. If minimal variations are detected between the new upload and an existing record, the original creator is credited as the source of inspiration.
  • Fostering Creativity:
    This feature ensures that creators are recognized for their originality while encouraging innovation within the community.

What We Learned

This project was an incredible opportunity for us as a team to expand our skills across various domains:

  • Real-Time Pose Detection on Device:
    We utilized MediaPipe for real-time pose estimation, ensuring all computations were performed directly on users' devices. This lightweight approach eliminated the need for server-side processing, enhancing both privacy and performance.

  • Modern Full-Stack Development:
    Using Next.js with TypeScript, we built a scalable architecture that tightly integrates the frontend and backend. This allowed us to maintain a highly interactive user experience while keeping the system efficient.

  • Database & Authentication Integration:
    We employed MongoDB Atlas as our database and Auth0 for authentication. One significant learning was synchronizing user creation across these systems—ensuring that when a user is created in Auth0, a corresponding entry is automatically generated in MongoDB using Auth0's unique key.

  • File Storage & Metadata Management:
    For storing large files such as videos and JSON data (representing pose information), we used Pinata. This setup, combined with our use of Pinata’s vectorized databases, allowed us to analyze metadata and track originality efficiently.

  • Advanced Learning Feedback Mechanisms:
    By deeply analyzing pose data, we developed a system that identifies misaligned body parts (nodes) during a dance routine. This intricate spatial and temporal analysis delivers actionable feedback to users, explaining what is wrong and how to adjust for improvement.

How We Built the Project

The development process was structured into several key stages:

Frontend Development

  • Framework & Language:
    Built with Next.js using TypeScript for its scalability and developer-friendly features.
  • Pose Detection:
    Integrated MediaPipe directly into the browser for real-time keypoint detection, ensuring calculations occur locally on the device.
  • UI/UX Design:
    Focused on creating an intuitive interface that supports both learning and sharing experiences.

Backend Development

  • Server-Side Logic in TypeScript:
    APIs were developed within Next.js to handle user authentication, video processing, and data management.
  • Authentication Integration:
    Auth0 was utilized for secure login and signup processes, synchronizing user accounts with MongoDB Atlas.
  • Data Storage & Processing:
    JSON pose data (generated from MediaPipe) and videos are stored on Pinata while references are maintained in MongoDB Atlas.

Challenges Faced

Developing this platform presented several challenges:

  1. Real-Time Pose Analysis Optimization:
    Ensuring MediaPipe ran efficiently across devices required extensive testing to strike the right balance between performance and accuracy.

  2. Synchronizing Authentication & Database Systems:
    Integrating Auth0 with MongoDB Atlas was complex. We had to design workflows that seamlessly tied user accounts across both systems while prioritizing security.

  3. Managing File Storage & Metadata Analysis:
    Storing large video files alongside JSON pose data in Pinata while utilizing its vectorized database for originality tracking demanded precise planning and execution.

  4. Creating Robust Embeddings for Originality Tracking:
    Initially, we faced challenges in developing effective embeddings from our dance data. After several iterations, we discovered that leveraging our comprehensive JSON pose data directly provided the consistency needed for Pinata’s vector database to accurately cluster similar dances and credit original creators.

  5. Detailed Feedback Mechanism Complexity:
    Developing a system that not only identified which nodes were off at specific timestamps but also provided actionable corrective guidance required advanced spatial and temporal analysis techniques.

Final Thoughts

This project exemplifies the powerful intersection of art and technology. By combining real-time pose detection with detailed feedback mechanisms, our platform offers a unique dance learning experience that highlights exactly which nodes are off at precise timestamps while providing bespoke corrective advice. At the same time, our innovative use of Pinata’s vector database—facilitated by JSON-based embeddings—ensures originality is tracked effectively so creators receive proper credit for their work.

Whether you’re honing your dance technique or showcasing your original moves, this platform empowers dancers to learn interactively while fostering creativity within an engaged community. We’re excited about how this blend of detailed learning feedback with advanced originality verification can transform traditional dance education into something more dynamic, accessible, and rewarding!

Built With

Share this project:

Updates