AR Pathology Assistant

Bridging radiology and gross pathology with AI-guided AR for precise specimen slicing.

Comment

Inspiration

This project was inspired by first-hand observations during one of our internships at a pathology lab, where we gained exposure to the workflow of specimen grossing. In this process, pathologist assistants (PAs) are tasked with localizing internal features, such as tumors, calcifications, or biopsy clips, from imaging and translating that information into physical cuts on surgical specimens.

We noticed that despite the availability of detailed 3D radiological scans, the alignment between imaging and the physical specimen is often manual, relying heavily on visual estimation and anatomical intuition, which becomes particularly challenging when features are small or non-palpable. Our team wondered: could we leverage computer vision and augmented reality to guide grossing with radiological precision?

What it does

Our prototype system assists PAs in accurately localizing and slicing through regions of interest, such as tumors or biopsy clips, by: 1) Uploading a CT scan of the resected specimen, 2) Allowing the user to define an optimal cutting plane on the 3D image, 3) Capturing the real-world surface mesh of the specimen using a LIDAR or mobile phone, 4) Aligning the virtual scan with the physical specimen using AI registration techniques, and 5) Projecting the selected cutting plane back onto the specimen in real time using an AR overlay. This provides intuitive, spatially-anchored guidance during grossing, helping ensure high-value tissue is sampled accurately.

How we built it

The AR Pathology Assistant was developed as a web-based augmented reality application, designed to be accessible on mobile devices without requiring specialized hardware.

We used AR.js for lightweight, marker-less AR tracking directly in the browser, coupled with WebRTC to access and stream real-time camera data. The frontend interface was developed with Vanilla JavaScript, HTML5, and CSS3, ensuring a responsive, mobile-first design.

Touch-based manual alignment tools were implemented for intuitive hologram positioning, while the backend logic simulates AI-driven registration algorithms for aligning the virtual model with the physical specimen. We hosted the application using Netlify Drop, allowing instant deployment with Progressive Web App (PWA) capabilities for offline access and device installation.

The system architecture is modular, supporting integration with medical imaging standards like DICOM and 3D mesh data for CT-based specimen processing. This makes the platform extensible toward full clinical utility.

Challenges we ran into

Building a clinically-relevant AR system in a browser environment introduced several technical and design challenges:

  • Real-Time Registration: Implementing an accurate alignment between physical and virtual models without depth sensors required both manual controls and a simulated AI pipeline.

  • Camera Calibration: Without dedicated calibration tools or hardware, achieving stable AR overlays was limited by variability in mobile camera parameters.

Accomplishments that we're proud of

  • Developing a functional proof-of-concept that demonstrates the feasibility of bridging radiological imaging with physical specimen guidance in real time.

What we learned

  • Details Matter: Precision in specimen orientation, margins, and user experience can dramatically affect usability and clinical outcomes.so we need to be very accurate if this wants to be used for clinical cases

  • Gaining hands-on experience with AR toolkits like AR.js and understanding the challenges of marker-less augmented reality in web browsers.

What's next for AR Pathology Assistant

Phase 1: Enhanced Detection and Automation

  • Implement automated specimen recognition and shape fitting
  • Expand support for different organ and specimen types
  • Integrate robust AI-based registration algorithms

Phase 2: Medical Integration and Validation

  • Enable DICOM file import and CT volume visualization
  • Build interfaces for real-time CT scan overlay on live specimens
  • Conduct usability testing with pathologists and assistants

Ultimately, our goal is to evolve the AR Pathology Assistant into a regulatory-grade decision support tool—enhancing precision, reducing variability, and shortening turnaround times in pathology labs.

Built With

Share this project:

Updates