Octavius

A team
A very cool robot arm
uwu

Inspiration

In the chaos of the battlefield, every second counts for a combat medic. With lives on the line, they must triage injuries, administer aid, and make split-second decisions - all while carrying heavy gear and navigating unpredictable threats. But what if they had an extra set of hands?

What it does

Octavius quickly detects injuries among soldiers in the battlefield and evaluates what next steps a combat medic should take. A robotic arm can also assist in common battlefield procedures like chest trauma and wound packing or simply holding instruments for medics. We showcase both these capabilities through abstract scenarios.

During evacuation, automated voice calls can also coordinate efforts among multiple combat medics. And finally, combat medics may access the medical history of all soldiers to make fast and informed decisions.

How we built it

Injury detection is done through the Apple Vision Pro and transmitted through Zoom. After transmission, the Jetson Orin Nano processes the real-time video feed using Nano Owl and sends instructions to the robot arm (that we fully build ourselves)! ElevenLabs is exclusively used for voice AI support and InterSystems is used for fast retrieval of all medical records through their vector embeddings.

Challenges we ran into

A major challenge we faced was obtaining the video feed for processing. The Apple Vision Pro has strict protocols around accessing this kind of data, so we had to get creative to tinker with it for this project.

On a very high level, we shared the Vision Pro view to a Macbook, then shared it via Zoom to a second laptop, from which we streamed it to the Jetson Nano via OBS RTSP. This process is documented in more detail in our repo’s readme.

Accomplishments that we're proud of

ROBOTIC ARM - we fully prototyped, built, and iterated on the robotic arm ourselves at Stanford PRL!

What we learned

Combat medics are heroes. This project further proved to us their importance and impact on any battlefield.

Secondly, Apple’s privacy protections are remarkable, especially when it comes to accessing the camera feed from the Vision Pro. We believe we were able to hack together the most optimal solution to access the camera for our use case (which is completely novel) and has under 1 second latency!

What's next for Octavius

We are proud to have built Octavius. Next, we plan on bringing more AR support for combat medics which in many cases will require Apple’s Enterprise Permissions. We also plan to explore the use of fully autonomous humanoid robots to work alongside combat medics.