Brainropto

The lecturer being tracked in a panopto video
The facial tracking that is active throughout the site
The settings page
An example of a gesture made my the user being picked up

Inspiration

We understand that students can often struggle to focus when catching up with lectures due to the lack of peer pressure to actually pay attention when catching up at home. We wanted to implement a range of attention capturing techniques in order to help out our fellow students. After observing a maths lecturer and often commenting to each other about how his ‘on one hand, on the other hand’ movement looked like the 6-7 movement, we were inspired to see if we could teach ourselves enough about motion tracking in 24 hours to bring our idea to life.

What it does

Our website takes in a link (can be Panopto or Youtube) or a video file and uses motion capture via MediaPipe in order to visualise the lecturer and their movements. With this data we can pick up some common movements such as 6-7 and the Rick Astley dance and display fun graphics when these are detected. We also created a log of these movements, allowing users to jump back to these hilarious moments.

We have some features that keep the users attention. For example, we have a loud metal pipe sound that goes off randomly every 3-10 minutes to keep you engaged, a feature that plays a loud “WAKE UP” when your eyes are closed for more than 5 seconds (can be increased or decreased), and memes that pop up with various poses from the user such as hands on head, open shocked mouth, holding up their index finger and shushing.

Finally, we also have the option within the site to add some attention grabbing videos that are commonly used on the side of online content that users are used to tuning into such as: Subway surfers gameplay Minecraft Parkour Hydraulic press videos GTA car videos

How we built it

This website is built with React, TypeScript, and Vite. TailwindCSS handles all the styling for that crispy meme layout. Real-time gesture and face/hand detection is powered by MediaPipe running in the browser (no backend!), with all settings and user preferences saved locally using LocalStorage. Audio and image overlays are triggered instantly based on webcam input, and the whole app is designed to be responsive, performant, and extremely cursed.

Challenges we ran into

Low quality panopto recordings make tracking difficult as hands are too small/pixelated, so our software that works on other videos ends up struggling. We believe that if we were able to download lectures as .mp4, this higher quality would drastically improve the performance of our software and as a result we added the option to upload an .mp4. We tested this by recording our own lectures in panopto, meaning they were downloadable, and had great success. If this functionality ever became available to students, it would be a great option for better accuracy of the software. We also wanted to add some features using the transcript, but we found that this was only downloadable as a .txt file to the user’s computer rather than a JSON requesting, and intercepting the transcript request was not feasible. Thus we decided not to use due to time constraints.

Accomplishments that we're proud of

We are proud of the final accuracy of our motion capture. Despite the fact it is not perfect, we spent lots of time fine tuning our models and ensuring that actions had to be very explicit before being picked up. E.g. swinging arms by side won’t pick up a rick roll. We are also proud of how usable and intuitive our software is. Despite providing a help page for anyone who is confused, we believe that our system is easy to navigate and has a clean UI.

What we learned

We decided to utilise media pipe for our motion capture, something none of the team had previous experience in. By doing this, we learned about the tradeoffs that exist between using different model complexities and varying hyperparameters. We learned how to use landmarks to determine what qualifies as certain actions and how to fine tune x and y position constraints to make this as accurate as possible. We also learnt about web deployment using Vercel, and domain configuration.

What's next for Brainropto

In the future we would be interested in improving our integration with panopto and also optimising our motion tracking models. Due to how panopto itself works, we would love to work with the software to gain access to .mv4 recordings as lectures and be able to download the transcripts as JSON files, meaning we could pursue the development of new features.