CareBot AI

CareBot Robot built using Arduino (motion control) + Raspberry PI(AI agent)
Web app with landing Page with conversational-ai agent
Featherless.ai + Tesseract OCR pdf/document QnA pipeline
EMR report generated as PDF from convo history of patients (also possible as JSON) - easy to integrate with external EMRs
Dashboard with real-time data and metrics of patients and robot
Patient Registry with patient history who interacted with Robot synced to webapp
Interaction logs with the agent

Inspiration

The inspiration behind CareBot AI came from the need to restore the human connection in clinical consultations. In modern healthcare, the "communication gap" created by complex medical jargon and the burden of manual documentation often leaves patients feeling overwhelmed and clinicians burnt out and understaffed. We wanted to develop a "robotic guardian" that acts as a diagnostic bridge blending real-time speech intelligence with personalized care to transform dense clinical data into clear, actionable insights that empower both the patient and the provider.

What it does

CareBot AI is an integrated hardware and software solution designed to bridge the gap between patient interaction and clinical documentation. It centers around a robotic vehicle, a "compassion bot", that engages in real-time dialogue with patients, capturing symptoms and queries through a conversational AI agent that updates a centralized medical dashboard. Simultaneously, clinicians interact with a web-based AI assistant to input progress notes and distill "mountains of data" into structured EMR-formatted JSON and PDF reports, complete with automated ICD-10 mapping and suggested treatments. By translating complex medical jargon into patient-friendly recovery plans, CareBot AI provides a seamless, bidirectional interface where both the robot and the web app empower caregivers and patients to communicate with total clarity.

How we built it

We built CareBot AI as a synergistic hardware and software ecosystem. The robotic vehicle is powered by an Arduino Mega for movement, obstacle avoidance, and person-following logic written in C++, while a Raspberry Pi 5 serves as the onboard "brain", running a real-time conversational AI agent with integrated microphone and speaker modules. The web interface was developed using Next.js and TypeScript with Tailwind CSS and shadcn/ui for a responsive, clinical-grade dashboard. We leveraged the ElevenLabs SDK for high-fidelity voice interaction and implemented a real-time data pipeline using WebSockets to sync the robot’s telemetry and patient metrics with the web app. To handle "mountains of data," we built an ensemble processing pipeline using Tesseract and PDF.js for document extraction, routing insights through Featherless.ai to run cloud-based LLMs that support the Pi’s localized processing, ensuring seamless multilingual TTS/STT and audit-ready EMR generation.

Challenges we ran into

AI Latency on the Pi: Optimizing the conversational-ai agent to run on the Raspberry Pi 5 without significant lag during microphone/speaker I/O proved to be a major technical hurdle.
Hardware-Software Sync: We struggled to stabilize WebSocket connections between the Raspberry Pi and the Next.js dashboard, ensuring the robot's "memory" and metrics updated in real-time.
Pipeline Integration: Manually routing the ElevenLabs SDK to work with our Tesseract-Featherless.ai ensemble pipeline was complex, specifically when trying to query document data through the voice agent.
Simulation Discrepancies: Transitioning from Tinkercad to the physical Arduino Mega required hours of debugging to fix movement and logic inconsistencies of the Robot.
Scope Management: As our first time combining embedded hardware with a AI web app, balancing mechanical stability with a polished UI was a significant learning curve.

Accomplishments that we're proud of

Multidisciplinary Synergy: We are incredibly proud of how our team bridged a significant technical divide; despite half of us having no prior hardware experience and the other half being new to software, we successfully collaborated to merge C++ robotics code with high-level web architecture.
Seamless Tech Integration: In less than 24 hours, we successfully integrated a complex stack involving Voice-AI, hardware robotics, and cloud-based LLMs into a unified, functional ecosystem that handles real-time conversational flows.
Production-Ready Vision: We developed a low-cost, fully functional caregiver assistant that is "clinic-ready". Our system doesn't just theorize; it actively performs medical triage, generates structured EMR reports, and bridges the gap between patient interaction and professional documentation.
Accessibility & Inclusion: Creating a truly multilingual interface that is equally intuitive for both elderly patients and busy clinicians is a significant milestone for us, ensuring the "robotic guardian" is accessible to everyone.

What we learned

Cross-Disciplinary Integration: With half the team new to Arduino hardware and others new to software development, we learned to bridge a significant "technical communication gap." This taught us how to effectively translate mechanical requirements into code.
Embedded AI & Latency: We gained deep insights into hardware-software integration, specifically the complexities of running real-time conversational AI on edge devices like the Raspberry Pi 5.
Patient-Centric Engineering: Beyond the technical specs, we learned the importance of designing for empathy. We realized that for a "robotic guardian" to be effective, it must be as engaging and intuitive for the patient as it is functional for the clinician.
Collaborative Problem Solving: The importance of collaboration, persistence, and adaptability when building something completely new in a short timeframe.

What's next for CareBot AI

Advanced Vital Integration: We plan to equip the robotic hardware with IoT sensors (like pulse oximeters and thermal cameras) to automatically feed real-time vitals directly into the AI diagnostic engine.
Direct EMR Partnerships: We intend to develop native plugins for major EMR systems like Epic and Cerner, allowing CareBot AI to push structured JSON data and PDF reports directly into official patient records with a single click.
Enhanced Navigation Logic: We will refine our "person-following" and obstacle avoidance algorithms using LiDAR to help the robot navigate complex hospital corridors and high-traffic hospice environments more naturally.