CareFlow

AI-driven recovery management system: automating post-surgery check-ins and visual timelines to boost medication adherence and cut hospital readmissions.

Comment

CareFlow

AI-Powered Recovery Management System that transforms complex post-surgery care plans into intelligent, actionable recovery journeys

Inspiration

Nearly 20% of patients are readmitted within 30 days of discharge, often due to medication non-adherence and missed follow-up care. This statistic isn't just a number—it represents thousands of patients facing preventable setbacks in their recovery journeys.

For us, this problem is personal. Both our moms have over 30 years of healthcare experience, witnessing countless avoidable readmissions caused by confusing instructions and lack of timely support. Their experiences inspired us to take action.

Current solutions like basic reminder apps or periodic nurse check-ins aren't enough to address the core problems: complex instructions delivered as static PDFs, lack of real-time guidance, and no way to adapt the care plan based on patient progress. Patients need a system that transforms those dense discharge packets into an intelligent companion that guides them through every step of their recovery.

CareFlow reimagines the post-surgery care experience by using AI to extract, organize, and deliver personalized care instructions through natural voice conversations, timely reminders, and visual progress tracking—creating a bridge between hospital discharge and successful recovery.

What it does

CareFlow transforms post-surgery discharge PDFs into an intelligent recovery management system with four key functions:

  1. Automated Care Plan Extraction: Our system analyzes discharge PDFs and automatically extracts structured data about medications, exercises, appointments, and special instructions—turning dense medical documents into actionable events.

  2. Interactive Recovery Timeline: Patients navigate their care journey through a visual timeline that displays upcoming medications, exercise requirements, and appointments, with the ability to mark items as completed.

  3. AI Voice Check-ins: Using ElevenLabs technology, CareFlow conducts natural voice conversations to remind patients about medications, check on symptoms, and answer questions about their care plan—all using the context from their specific discharge instructions.

  4. Smart Compliance Tracking: The system automatically updates patient compliance based on voice interactions and app engagement, providing visual progress indicators and adapting future reminders based on adherence patterns.

The system serves multiple stakeholders:

  • Patients receive timely guidance and reminders through familiar voice interactions
  • Caregivers gain visibility into compliance and potential issues
  • Healthcare providers receive structured compliance data and potential concerns

How we built it

CareFlow's architecture combines multiple technologies to create a seamless recovery management experience:

Frontend:

  • Built with React for responsive design that works across devices
  • Implemented an intuitive timeline interface using React-Vertical-Timeline
  • Created accessible UI components with Shadcn/UI
  • Developed compliance visualization with React-Circular-Progressbar

Backend:

  • Developed two specialized backend services:
    1. RAG Backend: Processes PDFs and creates embeddings for intelligent retrieval
    2. Voice Backend: Integrates with ElevenLabs and manages patient interactions
  • Built API endpoints with FastAPI for high performance
  • Utilized LlamaIndex for document processing and agent orchestration
  • Implemented PostgreSQL with pgvector extension via Supabase for both relational and vector data

AI Pipeline:

  • PDF Analysis: Used Llama4 to extract structured events from discharge instructions
  • Conversational Agent: Integrated Groq-LLaMA2 for context-aware patient interactions
  • Voice Interface: Implemented ElevenLabs for natural voice conversations
  • Computer Vision: Added Gemini Vision AI for medication adherence verification

Data Architecture:

  • Designed comprehensive schema with tables for patients, care plans, events, medications, and call logs
  • Implemented vector embeddings for semantic search across patient documents
  • Created an event system that tracks completion status and patient progress

Challenges we ran into

PDF Extraction Complexity:

  • Discharge documents vary widely in format and structure
  • Needed to build a robust extraction system that could handle inconsistent layouts
  • Required careful prompt engineering to ensure accurate identification of medications, dosages, and schedules

ElevenLabs Integration:

  • Creating a natural conversational flow that felt empathetic while remaining medically accurate
  • Implementing custom tool calls to enable the conversational agent to update event statuses
  • Managing conversation context to ensure appropriate responses to patient concerns

Event Synchronization:

  • Coordinating event updates between voice interactions and manual app inputs
  • Building a reliable system for tracking patient compliance across multiple interaction channels
  • Creating accurate progress metrics that reflect true adherence

Technical Integration:

  • Managing multiple AI services while maintaining reasonable latency
  • Creating a seamless experience across voice and visual interfaces
  • Building a secure system that handles sensitive medical information

Accomplishments that we're proud of

  1. Created a fully functional end-to-end system that processes real discharge PDFs and generates structured care plans
  2. Built an ElevenLabs voice agent that conducts natural, empathetic conversations while accurately tracking medication adherence
  3. Developed a visual medication verification system using Gemini Vision that can identify pill-taking moments
  4. Designed an intuitive interface that simplifies complex medical instructions for patients of all technical abilities
  5. Implemented a sophisticated backend that maintains context across multiple patient interactions
  6. Created a flexible architecture that can be extended to additional medical conditions and care protocols

What we learned

  • The importance of context in healthcare AI applications—generic responses aren't sufficient for medical interactions
  • How to effectively combine multiple AI modalities (text, voice, vision) into a cohesive experience
  • Methods for translating unstructured medical text into actionable, structured data
  • Techniques for maintaining conversation state across asynchronous interactions
  • The challenges of creating accessible interfaces for users with varying technical abilities, especially during recovery
  • How to design systems that gracefully handle edge cases and unexpected inputs in a medical context

What's next for CareFlow

Enhanced Recovery Intelligence:

  • Predictive analytics to identify potential compliance issues before they occur
  • Customized care plan adjustments based on patient progress and feedback
  • Integration with wearable devices for objective recovery metrics

Expanded Medical Coverage:

  • Support for additional surgery types and chronic condition management
  • Specialized modules for pediatric and geriatric care
  • Integration with pharmacy systems for medication refill management

Healthcare Provider Integration:

  • Direct communication channels between patients and healthcare team
  • Provider dashboards for monitoring multiple patients
  • Integration with electronic health record (EHR) systems

Advanced Interaction Modes:

  • Multi-language support for diverse patient populations
  • Caregiver management portal for family involvement
  • Offline mode for areas with limited connectivity

Scalability Improvements:

  • HIPAA-compliant cloud infrastructure for enterprise deployment
  • Performance optimizations for handling thousands of concurrent patients
  • Comprehensive API for integration with existing healthcare systems

Built With

  • elevenlabs
  • fastapi
  • gemini-vision
  • groq
  • llama-index
  • next.js
  • pgvector
  • react
  • shadcn-ui
  • supabase

Team Members

  • Anwar Mujeeb: Elevenlabs integration with Groq tools and gemini video analysis, Frontend
  • Russell Semsem: PDF to SQL for Supabase using Groq, VectorDB on Supabase , Frontend

Built With

Share this project:

Updates