VibePath by Meta Glasses

Inspiration

Visually impaired people often rely on white canes, guide dogs, or basic navigation apps. While these tools are helpful, they still leave major gaps in environmental awareness, especially in crowded indoor spaces, unfamiliar buildings, and situations with moving obstacles. Tools to aid the visually impaired have still not capitalized on tech or artificial intelligence.

We wanted to build something that could not only provide direction but also create a system that actively understands the environment, guides users safely, and provides access to caregiver support if needed.

That is VibePath, an AI-powered navigation assistant that combines smartphone vision, LiDAR depth sensing, spoken guidance, haptic feedback, and optional caregiver monitoring all in one, super low-cost.

What it does

VibePath helps visually impaired users navigate safely in real time.

The app uses the iPhone camera and LiDAR sensor to scan the environment, identify nearby obstacles, understand floor and path layouts, and determine safe walking directions.

It then provides spoken directions such as “turn left,” “backpack ahead", and "table 50 feet away." Users can also ask questions, request help, or start a caregiver session.

If a caregiver session is triggered, the backend generates a secure link that can be shared through messages. The caregiver can then open a live web viewer to see the navigation feed, direction transcript, and emergency status.

How we built it

We built VibePath as an iPhone app using Swift, SwiftUI, and Xcode.

The app uses:

ARKit and LiDAR for depth sensing
The iPhone camera for scene understanding
Speech synthesis and haptic feedback for guidance
Gemini AI for scene interpretation and question answering
Firebase and Firestore for backend session management
Daily.co WebRTC for caregiver streaming
HTML, CSS, and JavaScript for the caregiver web viewer

The navigation system works by converting camera and LiDAR input into a simplified occupancy map of safe and blocked areas. The app then chooses the safest nearby path and continuously updates guidance in real time.

Challenges we ran into

One of the biggest challenges was building the caregiver support system, especially the live video streaming through Daily.co.

At first, we were able to successfully generate secure caregiver session links, but the live video feed often would not appear correctly in the caregiver web viewer. We realized that streaming full-resolution, high-framerate video in real time was causing performance and connection issues.

To solve this, we lowered the video frame rate and reduced the overall stream quality, which made the connection much more stable and allowed the caregiver feed to load consistently.

What we learned

We learned how to

Use LiDAR and camera data together
Build occupancy maps from raw sensor data
Use AI for scene understanding
Design for accessibility and user trust
Build a real-time backend with Firebase and WebRTC
Combine voice, haptics, and navigation into a single experience

We also learned that accessibility projects require extensive testing and attention to detail, because even small delays or confusing instructions can make a big difference.