Vita

Re-envisioning healthcare through a humanistic lens by leveraging AI to boost patient satisfaction and reduce doctor burnout.

Comment

VITA is an AR-powered assistant for healthcare physicians that intelligently transcribes, displays, and acts upon doctor-patient interactions in real time.

Thank you to Intel, Hume AI, and Groq for making the crux of our project possible.

Inspiration

A prominent issue that plagues the healthcare industry, especially physicians, is the time spent on using electronic health record (EHR) systems, both for retrieving and logging information. In the delivery of healthcare, time is a vital and limited resource. It contributes to healthcare costs and acts as a key element in patient-doctor relationships. Physicians spend, on average, half of their days (49.2%) on EHR and desk work. In addition, nurse practitioners spend an average of 33% of their work periods on EHRs, and studies show that this high level of technology usage leads to extreme levels of burnout and motivation loss. As for physicians, their burnout with EHR use has been significantly researched; high EHR usage significantly reduces available time to spend with patients and other tasks. In a study conducted in 2020, physicians in the moderate and high EHR cluster reported more stress and lower satisfaction compared to the low EHR cluster. Even more, it was only within the high EHR cluster that time pressure was associated with significantly more burnout, dissatisfaction, and intent to leave.

What it does

Vita is a sophisticated healthcare software that integrates with smart glasses equipped with cameras, such as Ray-Ban Stories, Meta, Google Glass, and the upcoming Apple Vision Pro. This innovative program enables doctors to wear these glasses and identify patients through DeepFace facial recognition combined with computer vision (CV) and augmented reality (AR). Once a patient is identified, the glasses retrieve the patient's data from a FAISS index through a trained vectorization model, mapping it to an SQL database that includes visit history, diagnoses, medications, and more.

The data is then loaded into a local model running on Intel Developer Cloud. Using Retrieval-Augmented Generation (RAG), Vita powers a quantized LLaMA 3 model to provide doctors with a comprehensive briefing on the patient, displaying essential information through an AR interface. This system allows doctors to begin recording their interactions, eliminating the need for a clipboard or computer. The AI dynamically parses the interactions into real-time bullet points, which are displayed in AR and later summarized into a paragraph by a fine-tuned LLaMA 3 70B model. We utilize Hume AI to monitor patient emotional well-being, in hopes of seeing it improve with more visits over time as well.

How we built it

Building Vita was a multi-faceted endeavor, requiring a blend of cutting-edge technology and a deep understanding of healthcare needs. We started by researching the latest advancements in smart glasses and AI, focusing on facial recognition, computer vision, and AR. Our team conducted a literature review on the needs of doctors and physicians to identify their pain points and needs, ensuring that Vita would address real-world challenges.

We developed the software using a combination of DeepFace for facial recognition, FAISS for efficient data retrieval, and SQL for database management. The local model and RAG technology are hosted on Intel Developer Cloud, providing robust performance and scalability with the added bonus of speed with it being local inference. The AR interface was meticulously designed to be intuitive and unobtrusive, allowing doctors to focus on patient interactions. We utilized Groq for high-speed processing and intelligent batching to ensure seamless real-time updates.

Challenges we ran into

Throughout the development of Vita, we faced several challenges. Integrating multiple technologies, such as facial recognition, computer vision, and AR, required overcoming compatibility issues and ensuring smooth interoperability. Chaining together multiple models and agents that perform differnet tasks in a memory-efficient manner as well as acutely tuning our models to do everything from multi-facial recognition to function calling was difficult, to say the least. Additionally, creating an intuitive user interface that doctors could easily adopt without extensive training was a significant hurdle.

Accomplishments that we're proud of

We are immensely proud of the technological innovations we've achieved with Vita. We successfully built a dynamic real-time transcription summarizer, an entirely novel feature that revolutionizes the way medical interactions are documented. This cutting-edge technology enables the instantaneous conversion of doctor-patient conversations into organized bullet points, which are then synthesized into comprehensive summaries by our fine-tuned LLaMA 3 70B model.

Additionally, the advanced AI technologies and seamless integration with smart glasses demonstrate our expertise and commitment to pushing the boundaries of healthcare technology. We have constructed a robust pipeline that addresses the critical issue of physician burnout, an area that has seen little innovation until now. By leveraging technologies like DeepFace facial recognition, computer vision, AR, and RAG, we've created a sophisticated yet user-friendly application that stands as a testament to our team's dedication to solving real-world problems with state-of-the-art solutions.

What we learned

Developing Vita has been an enlightening journey. We learned the importance of close collaboration with healthcare professionals to create solutions that genuinely address their needs. The experience underscored the potential of AI and AR in revolutionizing healthcare, provided these technologies are implemented thoughtfully and securely. We also gained valuable insights into the challenges of integrating disparate technologies into a cohesive, user-friendly system.

What's next for Vita

The future of Vita is brimming with potential. We aim to expand compatibility with a broader range of smart glasses and enhance the AI's capabilities through continuous learning and improvement. We plan to incorporate additional functionalities, such as predictive analytics for patient outcomes and automated scheduling for follow-up appointments. Our ultimate goal is to make Vita an indispensable tool in healthcare, reducing burnout, enhancing patient care, and making healthcare more accessible to all.

Built With

Share this project:

Updates