Mistral-OUI

Contextual AI based on Vision and Summary in AR on a Magic Leap 2

Comment

Inspiration

The vision behind Mistral-OUI combines the innovative fields of Vision and Augmented Reality (AR) to revolutionize user interaction within their physical environment. Inspired by the potential to seamlessly blend digital information with the real world, our project aims to leverage AR technology to enhance user experiences in a multitude of contexts, enriching daily tasks and interactions with a layer of intuitive, contextual digital augmentation.

What it does

Mistral-OUI is an augmented reality application designed for Magic Leap 2 that intelligently recognizes the user's surroundings and dynamically generates a user interface (UI) tailored to their current context and situation. This spatial and contextual UI aims to significantly boost productivity and enhance the overall user experience by providing relevant, on-the-fly information and tools across a wide range of activities.

How we built it

The development of Mistral-OUI involved integrating advanced vision and AR technologies to create a contextually aware system capable of understanding and interacting with the user's environment in real-time. By utilizing Magic Leap 2's robust AR capabilities, we designed the app to analyze the spatial and contextual data of the user's surroundings and subsequently generate a customized UI that seamlessly integrates into their field of vision, offering interactive elements pertinent to their current needs. In the background, a two step process allows the experience to run

  1. The image is captured and described thanks to https://github.com/vikhyat/moondream
  2. Another API call to the Mistral API allows to create new html pages based on the context previously captured

Challenges we ran into

One of the main challenges we faced was rendering the html content on device because of permission issue on the web browser.

Accomplishments that we're proud of

We are particularly proud of successfully creating a highly adaptive and intuitive AR application that effectively enhances user productivity and experience through a contextually aware UI. The ability of Mistral-OUI to dynamically generate customized interfaces based on the user's environment marks a significant achievement in AR application development. Additionally, the positive feedback from initial users regarding the app's utility and innovation has been immensely gratifying.

What we learned

Throughout the development of Mistral-OUI, we gained valuable insights into the complexities of integrating vision and AR technologies to create a contextually aware application. We learned the importance of detailed environmental analysis and the challenges of designing a universally adaptive UI. The project also enhanced our understanding of the potential and limitations of current AR hardware, such as Magic Leap 2, in supporting advanced, real-time applications.

What's next for Mistral-OUI

Looking ahead, we plan to further refine Mistral-OUI by

  1. Adding XR Browser interactions.
  2. Solving browser permission issues.
  3. Collect videos and other format documents to convey the message of the project instantly.

You might see this page quite incomplete, stay tuned for further updates.

Built With

Share this project:

Updates