What it does
An AI tamagotchi to fix your horrible posture and remind you to touch grass. As CS students, we spend an embarrassingly large amount of time sitting hunched in front of desks. This is bad. Over 16 million Americans experience chronic back pain, contributing to approximately 83 million work-loss days. We thought about tackling more “serious” healthcare problems, but this is one that nearly all of us experience―and one that really does have an impact in daily life.
Tachigurasu has a built-in camera, a tiny round screen, and a swiveling servo base. It monitors your posture in real-time, and when it detects you slouching or leaning too far forwards, its animated virtual creature starts to die. It also detects how long you’ve been sitting, and nudges you to take a break.
There are other posture detecting apps out there. But onscreen popups and notifications can feel annoying and are easily ignored―clicking “dismiss” is almost a subconscious reaction at this point. Instead, we turn to real-world emotional manipulation. How could you let down a cute little pet that just wants some attention? nya. uwu.
How we built it
The hardware consists of an ESP32-cam with builtin OV640 image sensor, a round TFT display communicating over SPI, and a small SG90 servo, all stuffed inside a custom 3D-printed case designed in Onshape.
The inference can be done in the cloud by leveraging Modal’s serverless infrastructure, or entirely on the ESP32 using a TinyML model trained on a custom posture dataset. We can also offload the processing locally onto a laptop that communicates with the Tachigurasu. We process images with OpenCV, run Mediapipe pose and facial mesh detection, and use a hybrid heuristic/neural network system built with scikit-learn, Pytorch, and 300 images of Alex’s face for the end posture score.
Challenges we ran into
Hardware is hard. We had to rewrite graphics drivers in pure C because we were dissatisfied with the framerate of existing libraries. We burned an ESP32 by mixing up power and ground (oops). The ML portion is also surprisingly nontrivial―it needs to be computationally inexpensive while also being accurate.
Streaming video data from the ESP also turned out to be surprisingly hard. Because the ground plane of the ESP32-Cam is badly designed (this is what we get for buying $10 camera boards), there was significant interference in some of the I/O pins, causing its antenna to be weak. We ended up taping aluminum foil over the antenna in an attempt to boost its signal.
Accomplishments we’re proud of
Going from a box of random scrounged hardware to a real, tangible product in just thirty-six hours. Making an AI project that isn't yet-another-ChatGPT-wrapper. Conducting the worst soldering job known to mankind.
What we learned
Divide and conquer -- even as a two-person team, splitting tasks early on meant that we could develop features in parallel, greatly speeding up implementation time. Give up on dead ends -- we spent forever trying to get a certain graphics library to work faster; it turns out that particular one had a fundamental design flaw that could not be fixed. Start with a roadmap -- what's the minimum viable product (and the fastest way to get there?) What features would give a significant benefit? What components are "nice to have"?
What's next for Tachigurasu
We'd like to build into a fully independent system with LiPo batteries and full onboard processing. There's also a lot of work that can be done on making the movements and animations more fluid, for increased emotional manipulation appeal.
Log in or sign up for Devpost to join the conversation.