Automedic

Automedic
Showcase of All Patients
Page of Single Patient
Transcription Page
Sign In Page

💡 Inspiration

Doctors spend almost half their time on paperwork instead of patients. Information gets lost, notes are rushed, and patients leave without fully understanding what happened in the appointment. We asked: what if a small device could sit in the room, handle all of that automatically, and hand the doctor a clean, structured report before the patient even walks out? That's Automedic.

🩺 What It Does

Automedic is a smart diagnostic assistant that runs during a real patient-doctor consultation. Here's what it does all at once:

Listens to the conversation and converts it to text automatically Measures vitals, heart rate, blood oxygen, and heart rate variability using a wearable PPG sensor Analyzes what it sees using OpenCV and a VLM (llama3.2-v) to describe and flag visible conditions thoroughly. Runs a multi-agent AI system that thinks through the diagnosis the way a clinical team would Generates a structured EMR with diagnosis, care plan, ICD-10 codes, medications, and follow-up instructions Delivers that report to the patient instantly via email, one click, done

No manual notes. No lost information. No medical jargon that the patient can't understand.

🔧 How We Built It

Hardware: A MAX30102 PPG sensor reads heart rate, SpO2, and HRV metrics in real time over I2C A Bluetooth microphone captures the full conversation An ESP camera module feeds live images to a computer vision model A physical button lets the physician trigger image capture and visual diagnosis whenever they see fit

AI Pipeline — Multi-Agent System: Audio is transcribed using Google Speech-to-Text A LLaMA model summarizes the transcript, HRV vitals, and camera findings into a unified clinical summary Agent 1 — Triage Agent: Maps symptoms to an initial diagnosis the way a nurse would perform triage, using the NEWS2 score and chain-of-thought reasoning to arrive at a differential diagnosis with maximum accuracy Agent 2 — Care Plan Agent: Takes the triage output and generates a full care plan including ICD-10 code assignment, treatment recommendations, medications, and next visit scheduling All models run locally using Mistral 7B; patient data never leaves the device, protecting privacy Final output is compiled into a clean EMR entry

Backend: Everything streams to a Node.js backend over WebSockets in real time The final report is packaged and sent to the patient via email.

😤 Challenges We Ran Into

Honestly, this was harder than we expected. Lots of challenges, but also lots and lots of learning:

Noisy sensor data — raw PPG signals are messy. We had to write custom bandpass filters and peak detection just to get usable heart rate readings Bluetooth on Raspberry Pi — Getting stable audio recording over Bluetooth took way more debugging than it should. rfkill blocks, PulseAudio conflicts, PipeWire issues we hit all of it Microphone hardware limitations — midway through we realized our original microphone component produced audio too noisy to transcribe. We tried everything to fix it in software before finding a working solution Syncing three data streams at once — audio, biometrics, and video all running simultaneously without dropping data required careful async architecture PyTorch on ARM64 — our on-device model wouldn't run on the Pi's processor, which forced us to completely rethink our inference setup mid-hackathon Running LLMs locally — making Mistral 7B run reliably for real-time clinical inference while keeping latency low enough to be useful was a significant engineering challenge

🏆 Accomplishments We're Proud Of

It actually works with real sensor, real audio, real transcription, real multi-agent diagnosis, real report. No mocked data anywhere A single button press triggers the entire pipeline simultaneously — vitals, recording, camera, transcription, summarization, diagnosis, and delivery We turned a $35 Raspberry Pi into a functioning medical-grade edge device Our multi-agent system reasons through diagnosis, the way an actual clinical team does triage first, then care plan Patient data never leaves the room; everything runs locally on Mistral 7B The whole system works over a local network with no cloud dependency for the hardware layer

📚 What We Learned

Building with real hardware is a completely different world from building software; when something breaks, you can't just check the logs Noisy real-world data needs a lot of work before AI can do anything useful with it Async architecture matters enormously when you're running multiple data streams at once The hardest part of medical AI isn't the model, it's making sure the data going into it is clean and reliable Running AI locally for privacy is the right call for healthcare, and it's more achievable than most people think

🚀 What's Next for Automedic

Offline speech recognition using Whisper, so transcription also never leaves the room EMR integration via HL7 FHIR, so reports flow directly into hospital systems automatically Broader computer vision coverage means the camera can identify more conditions. Plug-and-use hardware kit, something any clinic can set up in minutes without technical help