Video Call Posture Checker

The basic interface of our web app

Inspiration

All of us tend to use our computers for rather extended periods of time and we noticed that it has had a somewhat detrimental effect on our posture. So we set out to design a web app that would help alleviate that problem! We felt that this program could also be applicable for digital life during the pandemic, as it could be used to help monitor one's posture during video conferences or calls.

What it does

It checks orientation of your body. So the feedback would be to turn left or right, so you face the screen. In addition to that, it will tell you how to center your face on the screen. After that, there is another button that gives feedback on your posture.

How we built it

We started by researching different pose recognition software, and chose tensorflow which can be used in an html document. We then set up Javascript to capture an image from webcam feed at 0.5 second intervals and analyze the image to collect certain data points about the users face and shoulders. This data is then used to determine whether the user is in the proper orientation. The code is contained in a single html file which we uploaded to Github so that it could be available on the internet as a static page.

Challenges we ran into

It was difficult to get the tensor flow model we were working with to work with a video. Originally it was meant to detect body parts using static images, so every time we tried to run it with video it would not provide any usable information. What we ended up doing instead was taking snapshots of the video feed at regular intervals and running those images through the tensor flow model.

Similarly, we were unfamiliar with the model we were using and learning how to extract the specific pieces of information we wanted from the images proved difficult at first.

On the front end side of things, it was Justin's first time working with html/css, so they had to learn it from scratch

Accomplishments that we're proud of

We were proud that we were able to get the video feed to work with the tensorflow model we were working with, and that we were able to pull the relevant pieces of data that we needed from the program. Also, this was the first hackathon for all of us, so we're really happy that we were actually able to complete a project in such a short time frame!

What we learned

Justin

I learned significantly about javascript, html, and css which are all languages that I have had no experience with previously! Plus, I got exposed to some computer imaging which I felt was a super cool experience in itself! Kim
I expanded on my knowledge of javascript, HTML/CSS, which I had some experience in before. I thought it was cool seeing how you can take some code from github and export it in order to make your project work because I have never done that before. After that, I think it was cool working out the kinks of how to use the computer vision to see your orientation and posture, and I like how its application can be useful for the future. Flepp
I learned more about using image recognition software and using preexisting machine learning packages to analyze images. I also learned more about Javascript and html.

What's next for Video Call Posture Checker

We wanted to add an overlay on top of the video feed that would display visual cues (such as arrows) that correspond to the feedback text.
Fix some minor bugs. Currently both functions will run simultaneously if the posture "check button" is pushed after the "orientation check" button. A refresh in between button clicks is needed to run each function separately.
A variant of the app that will check your posture from a point of view perpendicular from where you are facing, to see if your back posture is good
Calibrate it better so it can more accurately identify poor posture form