HoloHuman XR

Hero Banner
System Architecture

Inspiration

According to the WHO: Cardiovascular disease is the #1 killer in the world. 80% of all heart disease deaths occur in developing countries. Even in the U.S., rural patients are 1.5x more likely to die. The common cause? A massive gap in specialist access, forcing less-experienced doctors to make critical decisions without expert support. Until now.

That’s why we built a VR toolkit that brings expert insight directly to the point of care - letting clinicians peel apart life-sized anatomy, run real-time AI ECG analysis, and step inside medical scans with haptic feedback you can feel. And with live multi-user collaboration, experts can join and provide specialist advice on the same patient instantly, from anywhere in the world.

What it does

Peelable, life-sized anatomy: Toggle the visibility of skin, muscles, skeleton, and organs. You can grab, rotate, and scale the model with VR controllers.
AI ECG analysis (6 conditions): A TensorFlow/Keras model provides real-time predictions with confidence scores for: 1st-degree AV block; Right/Left Bundle Branch Block (RBBB/LBBB); sinus bradycardia/tachycardia; and atrial fibrillation.
3D medical imaging viewer: Load X-rays and CT scans in VR, navigate slices, and use zoom/pan. There’s a “portal” effect so you can “dive into” the scan space.
Haptics: Heartbeat pulses synchronized to 60–120 BPM and a “fracture” sensation when you touch broken bones, using Quest controller vibration.

How we built it

VR app (Frontend): Unity 2022.3 LTS with the Unity XR Interaction Toolkit, OpenXR Plugin for Meta Quest 2, C# scripting, and TextMeshPro UI.
AI/ML backend: Python 3.8+ with a Flask REST API that serves a pre-trained TensorFlow 2.2/Keras ECG model (antonior92/automatic-ecg-diagnosis). DICOM medical imaging samples are converted to PNG for VR viewing.
Hardware: Meta Quest 2 (with Quest Link over USB-C for development).
Assets: VR-optimized 3D anatomy models and sample medical imaging (DICOM → PNG).
Repo layout & docs: A UnityProject (VR app), a Backend (Flask API), and extensive docs.
Not only that, Meshy eliminates the time and cost barriers of traditional 3D printing by producing high-fidelity VR prototypes of cardiovascular devices, bioprinted tissues, and transplant planning models - creating a shared immersive workspace for medical teams worldwide.

Challenges we ran into

Keeping frame rate high on Quest 2 while rendering detailed models.
Wiring Unity to a local Flask server (CORS, data formats, latency).
Getting anatomy layers to align and feel natural to “peel.”
Preparing CT/X-ray slices and navigation.
Tuning haptics so they feel realistic, not annoying.

Accomplishments that we're proud of

A well-organized public GitHub repo (MIT license) with clear structure: Unity VR app folder, Backend API folder, and a references/resources area.
Thorough, builder-friendly documentation: a complete resources list, a comprehensive 36-hour development plan, an architecture/context doc, and a detailed task checklist with 165+ items.
A defined tech stack and integration points so others can pick it up and continue (Unity + OpenXR on Quest 2, Flask + TensorFlow/Keras ECG model, DICOM → PNG workflow).

What we learned

Unity XR Interaction Toolkit with OpenXR on Quest 2 for VR interactions and layer toggling.
Serving a pre-trained ML model via a lightweight Flask REST API (TensorFlow/Keras).
Preparing medical data for VR by converting DICOM samples into PNG slices and enabling slice navigation.

What's next for HoloHuman XR

MVP goals: Quest-ready VR scene; peelable heart/anatomy; ECG visualization with AI analysis; basic imaging viewer; controller haptics.
Stretch goals: Full skeleton model; hand tracking; portal effect; multiple ECG analysis modes; voice commands.
Future vision: Real-time ECG from wearables; patient-specific anatomical models from DICOM segmentation; multi-user collaboration (clinicians + patients); AI-powered automatic anomaly detection; hospital PACS integration; and working toward FDA-cleared medical device classification.