CAT | Devpost

CAT Logo
Dashboard view example
Sensor board
CAT application flowchart
RPi4 with CAT alert

Inspiration

Healthcare is something that everyone needs, but not everyone can easily get. One of our lead designers was inspired to create CAT when their family was forced through a difficult time with a loved one’s health. The difficulties of caring for elderly family members, worrying about the mental health of daughters and sons, and even the personal struggle of anxiety are things that everyone experiences. From financial cost to opportunity cost, the price of healthcare services, including mental health services, can be too much for many people in the world. Through these truths and personal experiences we came up with the idea for Computer Assisted Therapy (CAT).

What it does

CAT is an all-encompassing health oriented smart watch which uses various sensors to collect health data about its wearer including: ambient temperature, volatile organic compounds in the air, surrounding CO2 levels, fall/trip detection, and simulate heart rate detection. From this collected data CAT will communicate with its wearer through whatever concerns, questions, or emergencies they may have. While being great for monitoring one’s physical health, CAT is also able to assist someone through their mental health struggles. Using OpenAI’s WhisperAI and ChatGPT, CAT can listen to and respond to the wearer with answers to anything they ask. Along with this CAT also aims to provide natural conversation to its user. As opposed to commanding CAT with something like “hey CAT” to activate its listening, the aim is to move CAT close to your mouth to speak then activating its listening. CAT aims to provide friendly help and seem more like a pet rather than a servant like most AI assistants. Along with using the collected data to accurately inform the user of their health, CAT will also send a report of the wearer’s health to the CATDash terminal which allows for caregivers and guardians to easily view the health of all CATs in their litter.

How we built it

CAT is built with a combination of hardware, web, and AI technologies. Our hardware consists of the CAT wearable and an additional sensor package. The CAT Wearable has a custom 3D printed case that allows it to be worn as a watch and an LCD screen to show a quick status of the user's health. Because the LCD and case blocks the RPi4 IO ports, a separate device utilizing an ESP32 microcontroller and various sensors detects environmental and health data including temperature, air quality, and acceleration (falling). This information is sent to a main web server which runs a MySQL database and Apache web server to display event logs on our "CATDash" web dashboard. PHP is used to provide backend and API services between the various devices. WiFi and HTTP POST and GET requests transmit information between these three devices wirelessly and seamlessly. CAT uses this information as well as user voice input to provide assistance and companionship using OpenAI's ChatGPT and whisper machine learning models to accurately capture and respond to user microphone input. CAT will then send ChatGPT's response to our Coqui text to speech program using python sockets. From here the output is played back to the user as audio which they can listen to and continue their conversation with.

Challenges we ran into

One challenge we ran into was determining how to connect our devices. We were unfamiliar with connecting devices without an existing wireless or wired network in a secure manner. We were able to utilize a Windows feature that creates a mobile hotspot that we adapted into an ad-hoc network.

One major challenge for the implementation of AI was ensuring smooth transfer of data between the sensors, servers, and external APIs. Utilizing HTTP requests provided us the means to send data between our devices in order to facilitate this.

Another major challenge was implementing Coqui as this was a new tool that we had not used before. The text-to-speech processing could not run under Windows and we had run out of physical computers. An Ubuntu virtual machine on our private network was able to run this new library.

Finally, we ran into issues relating to incompatibilities between sensors and microcontrollers. We had hoped to include an ultrasonic sensor for activating the microphone, but it required a 5V source. Since the ESP32 outputs 3.3V, we improvised a solution utilizing a button.

Accomplishments that we're proud of

The thing we are most proud of with CAT is how we accomplished connecting so many moving pieces involving hardware, backend, web, and AI. This task initially seemed daunting as none of us had worked on a project of this scale. We were able to connect three devices wirelessly through a network/web backend that provides actual sensor data for communication to our users visually and audibly.

We are also proud of our custom 3D printed cases that complement our physical devices. Creating these cases was a difficult and iterative process that required a lot of prototyping. Their creation perfectly suits our use and enhances the usability of our design.

What we learned

We learned many new technologies and expanded our knowledge of ones we used previously. First, our speech implementation utilizing Coqui was entirely new to us. From using this we learned how to use quality text to speech models allowing for an audio output where otherwise there would only be text teaching us how to give users a more immersive and natural experience.

This also required us to learn how to send information between separate Python programs by opening a TCP socket to listen for data. This allowed us to send text from our whisper model to the text-to-speech model for output to a speaker. As an aside, we see possible applications in creating our own server applications through Python instead of relying on Apache/PHP.

We also learned new ways to better format our front-end web applications. We used the bootstrap CSS libraries for the first time, providing a cleaner, more user friendly experience than we have before.

What's next for CAT

In the future, we would like to incorporate more safety and health related information. This includes GPS, emergency keywords, better quality sensors, and gunshot detection from one of our previous projects along with SMS reporting to make alerts more accessible. Finally, by adding additional machine learning models, we could track and alert health concerns even better.