MaxWell

Inspiration

While medical malpractice has been a growing concern in the past few decades, doctors and healthcare professionals play a more important role than ever in ensuring patient safety and health outcomes. Unfortunately, access to medical resources is often limited, and for less serious injuries, patients often have to treat themselves despite having no previous medical knowledge or experience.

This often leads to patients making common mistakes in using home remedies, such as applying petroleum jelly to burns, that worsen the situation. We hope to build an all-in-one AI and Augmented Reality First-Aid Kit to address these issues of medical inaccessibility and limited general public medical knowledge in promoting a safer, more consumer-focused approach to patient safety.

What it does

Our goal was to provide a real-time interface to instruct and guide users on how to address minor injuries, either for themselves or others. MaxWell does this in a few ways: first, for well-known injuries like burns, a visual and interactive step-by-step tutorial on how to address the injury is provided in the AR view. Secondly, an AI-powered voice assistant exists to provide someone for patients to consult with when tending to their injuries in case they have any questions.

How we built it

Our final project consisted of two key components: an Augmented Reality (AR) user experience and real-time voice assistance.

The AR user experience was developed using A-Frame and AR.js, which provided AR capabilities in the browser. We deployed this experience on the Meta Quest 3 to take advantage of its advanced mixed reality features, including high-resolution passthrough and enhanced depth-sensing capabilities. The experience allowed users to interact with digital overlays, creating an intuitive and immersive environment that felt natural within physical space, especially under high-stress environments, such as treating first-aid wounds for your loved one.

The real-time voice assistance was designed to provide users with instant feedback and support through natural language processing. This component leveraged OpenAI's real-time API to enable conversational responses and voice interaction. We used websockets to handle bidirectional communication between the AR client and our backend, allowing real-time input and output. Users could issue voice commands to request information, with responses processed and delivered instantaneously.

The backend infrastructure was hosted on AWS, where we used AWS CloudFormation to manage and automate the deployment of resources, letting us maintain infrastructure as code for a scalable and easily repeatable system. Specifically, CloudFront is a global CDN that caches and distributes our Next.js app at the edge for better performance, CodeBuild is the AWS service that builds and packages our Next.js application into a Docker container using an EC2 builder instance, ECR stores the resulting Docker images, ECS runs and orchestrates these containerized Next.js applications, and S3 can store static files or build artifacts.

Challenges we ran into

Our main challenge was integrating all the different forms of multi-modal inputs into a single app compatible with extremely new tech -- in this case, Augmented Reality. For instance, as newcomers in the AR space, we spent many hours attempting to even get a button to appear and be clickable in AR, let alone overlaying a boundary box localized to a person's hand.

Accomplishments that we're proud of

One of our biggest accomplishments was successfully developing an AI-powered health assistant in XR despite most of our team having no prior experience with AFrame, WebXR, or building for Meta Quest 3. The learning curve for developing in 3D was steep, from understanding how to render interactive UI elements in AR to handling real-time AI responses in an immersive environment. Yet, we managed to create a fully functional AI-first aid assistant that integrates multiple technologies seamlessly.

Beyond the technical achievements, one of the most valuable aspects of this project was deepening our understanding of the healthcare industry and the patient safety challenges facing modern healthcare. While we initially focused on providing AI-guided assistance for simple injuries like cuts and burns, we realized that the same approach could be expanded to help prevent medication errors, procedural and surgical mistakes, errors during routine patient care (such as pressure ulcers, blood clots, and falls), infections, and diagnostic safety issues. This broadened our perspective on the role of AI and AR in improving healthcare accessibility and reducing preventable harm.

On the technical side, we leveraged AWS to host real-time WebSocket support for AI responses, ensuring that our AI assistant could deliver fast, low-latency, real-time health guidance. Additionally, we integrated OpenAI’s API to provide intelligent, dynamic, and context-aware health recommendations, making the experience feel both personalized and responsive.

Our WebXR-based architecture, built with JavaScript frameworks, allows MaxWell to be cross-platform and scalable, making it accessible on modern XR devices like the Meta Quest 3. Beyond pushing the boundaries of Augmented Reality and AI in healthcare, we're incredibly proud of how MaxWell has the potential to empower individuals with life-saving knowledge while also addressing critical gaps in patient safety and medical accessibility.

What we learned

Our main lesson is that developing with Augmented Reality is much more challenging than expected, and that trying to prototype too extensively before developing in a hackathon setting can be counterproductive.

What's next for MaxWell

Our main goal for MaxWell is twofold: first, we hope to expand the capabilities of its injury detection and first-aid help beyond basic burn and cut detection. Secondly, we aim to expand the technical accessibility of MaxWell. We didn't have the time or resources to test this out yet, but we hope to eventually port MaxWell into a much more compact form factor, such as with Meta RayBans, to increase how practically and quickly MaxWell can be applied in a real-world setting.

Built With

• aframe
• amazon-web-services
• javascript
• openai
• websockets
• webxr