SimPatient

Login Screen
Main UI
Analytics Screen
Feedback Summary
Conversation Screen
Voice Screen

Inspiration

The inspiration for SimPatient came from witnessing the anxiety and uncertainty medical students face during their first patient encounters. We realized that aspiring medical professionals have limited opportunities to practice clinical consultations in a safe, judgment-free environment before facing real patients. So we envisioned a platform where future doctors could practice patient consultations anytime, anywhere.

What it does

SimPatient is an AI-powered clinical training platform that simulates realistic patient consultations for medical education. The platform allows medical students and healthcare professionals to conduct interactive patient interviews where our AI patient responds naturally with symptoms, medical history, and concerns, just like a real consultation.

How we built it

We built SimPatient by combining cutting-edge AI technologies with robust cloud infrastructure. The core of our platform is a fine-tuned Gemini Flash model, trained on HIPAA-compliant doctor-patient conversation datasets to ensure realistic and clinically accurate interactions. To create a truly immersive experience, we integrated ElevenLabs' voice AI for seamless voice-to-text and text-to-voice capabilities, allowing natural spoken conversations with our AI patient. All consultation practice data and user progress are securely stored in Google Firestore, providing real-time synchronization, scalability and for performance evaluation. We optimized the architecture to achieve sub-500 millisecond response times through streaming technology, making conversations feel natural and immediate.

Challenges we ran into

Our biggest challenges centered around achieving production-quality AI performance. Fine-tuning the Gemini model required extensive data preprocessing, we had to transform raw medical dialogues into one of the required format (JSONL) where the AI correctly assumes the patient role while maintaining clinical accuracy and conversational flow. Integrating ElevenLabs' voice AI added another layer of complexity, requiring careful synchronization between speech recognition, model inference, and voice synthesis.

Accomplishments that we're proud of

We're incredibly proud of delivering a fully functional, end-to-end clinical training platform that actually works in real-time. This achievement wouldn't have been possible without our team's dedication, collaboration, and collective problem-solving. Our standout accomplishments include successfully fine-tuning a medical AI model that responds with clinical accuracy and human-like conversational flow, and seamlessly integrating voice interaction capabilities that make the experience feel like talking to a real patient.

What we learned

This project was an intensive learning journey in AI integration and healthcare technology. We gained deep expertise in Google's AI ecosystem, from leveraging Gemini APIs and Google AI Studio for model fine-tuning to deploying production models on Vertex AI with optimized inference pipelines. We learned the intricacies of prompt engineering for medical contexts, ensuring our AI patient responds appropriately across diverse clinical scenarios. Working with ElevenLabs taught us how to build seamless voice interfaces, handling the complexities of real-time speech-to-text processing, managing audio streaming, and synchronizing voice synthesis with AI responses.

What's next for SimPatient

SimPatient's roadmap is ambitious and focused on maximizing educational impact. Our immediate priority is expanding our clinical scenario library to cover diverse specialties from emergency medicine to pediatrics, giving learners practice across the full spectrum of medical encounters. We plan to implement adaptive learning algorithms that personalize difficulty based on individual performance, creating customized learning paths for each user. We're exploring multi-modal capabilities, including the ability to interpret visual cues and physical examination findings to create even more realistic simulations.