RealityBridge - Copy Paste VR/AR Experience

Quickly prototype Virtual Reality scenes using AI Object Detection on the Qualcomm RB3 Gen 2 board.

Comment

Inspiration

Our project was inspired by the desire to quickly prototype scenes in virtual reality. This product will help with rapidly laying out models in a virtual 3D space for envisioning future projects and environments.

What it does

Our ergonomic copy paste handheld tool allows the user to scan objects in the real world, detect and classify the scanned object, and send it to the virtual reality scene.

  • 📷Qualcomm RB3 Gen 2 board scans the physical space like a handheld camera
  • When the user presses the copy button, a serial message is sent from the ESP32 to the Qualcomm (USB-C to USB-A, baudrate 115200)
  • 📶The Qualcomm receives the copy signal and publishes the latest detected high-confidence object to the Redis server
  • 📚When the user presses the paste button, the signal goes through the Qualcomm and into the Redis Server
  • The VR Unity scene is subscribed to the Redis server and listens for the user's button presses
  • 📦When the paste signal is recieved, the Unity project takes the AI identified name and uses the SketchFab API to spawn a 3D model in Virtual Reality
  • 🎨This allows the user to rapidly prototype ideas in virtual reality

How we built it

  • 3D Printed handle, laser cut acrylic mounting platform for the ESP32
  • ESP32 connected to hardware triggers and switches through GPIO pins

Challenges we ran into

  • GPIO pins not working on the Qualcomm board
  • Learning curve for using Qualcomm device, Redis Server, GStreamer pipeline
  • Lack of documentation for Qualcomm
  • No Python libraries, missing many linux tools on the Qualcomm device
  • Unable to parse mouse input data without python libraries, we tried listening to /dev/input/event4 but could not distinguish between the various button inputs on the mouse

Accomplishments that we're proud of

  • Building an ergonomic, cool handheld hardware unit
  • Set up a data server to distribute data between the components of our product
  • Learned new

What we learned

  • Learned about Serial connections between two embedded devices
  • Networking with an embedded device
  • Learned how to setup a Redis server for distributing information between many devices
  • Learned how to prototype in C and Python on an embedded device

What's next for Copy Paste VR/AR Experience

  • Improve the pipeline between the Qualcomm AI device and our custom made hardware peripherals.

Built With

Share this project:

Updates