ClinAIssist

Inspiration

After attending the Patient Safety Technology Challenge, we discovered that medical errors cause over 200,000 deaths annually in the U.S. Our research revealed that healthcare professionals often struggle to quickly identify and respond to alarms, leading to stressful, complex procedures. Inspired by SnapAR's Spectacles 5, we saw the potential of AR to reduce stress and data overload in high-pressure situations. Our goal was to create an assistant that helps healthcare professionals manage alarms and procedures, reducing errors and ultimately saving lives.

What it Does

ClinAIssist assists nurses and other healthcare professionals by offering the following features:

Alarm Notifications: Alerts healthcare professionals through SnapAR Spectacles when any alarms go off in the hospital, displaying the source of the alarm for quick identification
Step-by-Step Checklist: Provides a detailed checklist outlining the necessary steps to resolve the medical issue, ensuring nothing is missed during high-pressure situations
Medicine Verification: Scans and verifies the text on administered medicine to ensure the correct treatment is being given for the specific condition, preventing very common incorrect medication administration errors
Injection Assistance: Visualizes needle location within the body, as well as the depth measurement for all types of injections, aiding practitioners in administering injections with precision.

How We Built It

We structured ClinAIssist into multiple sub-projects, each contributing essential functionality to the Spectacles and ultimately alleviating stress for healthcare practitioners:

Alarm Notifications: We integrated with hospital central monitoring systems to track alarm statuses, storing this information in a MongoDB Atlas database. The Spectacles continuously fetch alarm data and alert healthcare professionals through on-screen messages and text-to-speech notifications, indicating the source of the alarm.
Step-by-Step Checklist: An interactive checklist is displayed based on the patient's specific medical procedure. Healthcare professionals can utilize hand gestures, detected by the Spectacles, to mark tasks as completed, streamlining their workflow and ensuring that no task is missed.
Medicine Verification: For medication administration, we employ Optical Character Recognition (OCR) which then uses NLP to ensure the correct medication is given to the patient. The system cross-references the administered medicine with the appropriate list for the procedure, enhancing patient safety.
Injection Assistance: When administering injections, we utilize world raycasting hit query techniques to project a virtual needle onto the patient’s 3D mesh model. This system accurately measures the needle's penetration depth into the body using vector displacement, aiding precise injection placement.

Challenges we ran into

Faced difficulties using Optical Character Recognition (OCR) due to outdated documentation from SnapAR
A recent update to Lens Studio introduced performance issues, causing the application to be slow and buggy at times
We encountered compilation issues with TypeScript that delayed our progress
Fetch requests from the alarm database are updated asynchronously while JavaScript operates synchronously

Accomplishments that we're proud of

We developed a robust alarm detection system that ensures timely identification of patient issues, enhancing response times
We successfully implemented interactive checklists that promote accountability and efficiency among nurses, significantly reducing the risk of mistakes in a high-stress environment
Our system accurately reads and verifies the labels on administered medications, ensuring that the correct treatment is provided to patients
We achieved precise visualization of needle placement and depth, aiding healthcare professionals in administering injections safely
We created multiple sub-projects that integrate with one another, resulting in a cohesive final product that enhances patient care and ensures patient safety

What we learned

We gained deep insight into what it's like to develop an Augmented Reality project, as well as the potential limitations present in such systems. We also learned the power of AI and how it can guide people through stressful tasks, specifically in the healthcare industry. The lack of accurate documentation was extremely daunting to us at first, but we got through it by not being scared to ask the on-site mentors for help.

What's next for ClinAIssist

While these were just features that we could implement in 36 hours, there are many more that we plan on implementing in the future, including but not limited to the following:

Training an ML model to detect exactly where on the patient's body an injection should occur
Creating a more expansive list of medical procedures with the help of doctors and health professionals
Storing patient data and checking whether or not they are allergic to any medication, ensuring not to administer anything that could harm the patient
Using GPS to identify which healthcare professionals are closest to the patient with the alarm and broadcasting the alarm message to many people at the same time
Possibly expanding to different AR/VR devices like the Meta Smart Glasses or Apple Vision Pro