Calisthenics Tracker

Making fitness safer and more inclusive by using ML to fix bad posture with custom suggestions. Fitness Exercises are great, but they can ineffective and injurious if performed incorrectly!

Comment

Inspiration

"Fitness exercises are very beneficial to personal health and fitness; however, they can also be ineffective and potentially dangerous if performed incorrectly by the user. Fitness exercises are very beneficial to personal health and fitness; however, they can also be ineffective and potentially dangerous if performed incorrectly by the user. Exercise mistakes are made when the user does not use the proper form, or pose." - https://arxiv.org/abs/2006.11718

With the advent of Covid-19, this problem has become even more exaggerated with a lot of people starting their own fitness routines but unable to seek proper advice from fitness trainers, paving the way for serious injuries.

Thus, we set our sights on building an application that would bridge this gap by providing personalized feedback to users based on their posture.

What it does

Our Application uses computer vision to analyze user uploaded videos of their workout and then, deliver specific insights to users by comparing their posture to "good" workout routines.

Specifically, we use key point estimation with the open source project, Open Pose, to track a sequence of joints during a video with a pretrained CNN (convolutional neural network). Then, we combine these vectors and analyze them to provide custom feedback based on the open source project, Pose Trainer. With Open Pose, we track over 25 different joints in the user's body to deliver directed insights! With our front-end, users can also record videos of themselves, leading to a fully immersive application and we also started a complete data pipeline on google cloud.

How I built it

Frontend

The frontend was built using HTML, CSS, and a lot of JavaScript!

Backend

The backend was built using Python Flask to deliver the ML.

Machine Learning

We used keypoint estimation with a pretrained CNN on OpenPose to keep track of 25 different joints in the user's body. Then, we combined these vectors to get a sequence of positions which we compare with our good workout routines and their characteristics (like having your hands stable during a pushup).

Google Cloud

Our Architecture consists of the use of Cloud Storage, Pub-Sub, and Compute Engine. The front-end uploads a video to cloud storage, which triggers a notification to the pub sub topic and we have a subscription on our compute engine instance which triggers the ML model (which downloads the video, and delivers a prediction).

Challenges I ran into

  • Connecting the ML to the frontend
  • Setting up the google cloud

Accomplishments that I'm proud of

  • We built a complete application with ML and on the cloud, all within the span of less than 36 hours!

What's next for Calisthenics Tracker

We want to add the sendgrid room api to create a live streaming application and analyze live videos rather than recorded ones.

Share this project:

Updates