[C04] waitAR

XR assistance and on-the-job training platform that uses XR and AI to transform new restaurant staff into experienced and knowledgeable dining and hospitality specialists from day one.

Comment

✨Inspiration

Both Till - the lead engineer and Tim - the PM were waiting tables when they were teens. Both share the dreaded memories of being clueless and scared teens faced with meeting hungry strangers in a stressful and time-pressured environment and being expected to instantly know where the correct table is, who ordered what, what is the daily special, what are the vegan meals and what wine goes with the halibut filet.

Both Tim and Till decided that they owed their former selves this redemption - and need to give them and every other newbie waiter the superpower – to know all that and more before even learning anything of it.

🍷What it does

waitAR is an XR assistance and on-the-job training platform that uses XR and AI to transform new restaurant hosting staff into experienced and knowledgeable dining and hospitality specialists from day one, dramatically reducing onboarding time and elevating the guest dining experience.

waitAR is an intuitive, heads-up display system that provides real-time, contextual guidance directly in the user's field of view. The waiter sees the real world overlaid with a layer of actionable intelligence, making them feel confident, knowledgeable, and efficient. waitAR uses advanced NLP integration to process, analyze and act on the voice input from the waiter (wearing the headset) and the guests.

Features Present timely and relevant hospitality hints: welcoming the guests, offering a recommendations summary, ask for their preferences and dietary restrictions

Show the recommended dishes, wine pairings

Allow to take the order while maintaining eye contact. AI NLP listens to the guest and constructs an oder that can be processed by the kitchen

Anchor the order to the table - navigate the dish delivery to the correct guest.

💰Business case

The Problem: The Hospitality Skills Gap The restaurant industry is facing a critical challenge defined by four key issues: High Staff Turnover: Training new, inexperienced staff is costly and time-consuming. Leading to constant, expensive, and time-consuming training cycles. Inconsistent Service Quality: New staff cannot deliver the same level of service as experienced veterans, affecting the guest experience. Lost Revenue: Inexperienced employees often miss key opportunities to upsell high-margin items or recommend specials. Cognitive Overload: New hires are overwhelmed trying to learn the floor plan, menu, and procedures all at once, leading to stress and errors. The Solution: The waitAR Advantage waitAR transforms a novice into an expert from their very first shift by focusing on four principles: Context is Key Heads-up, real-time contextual guidance delivered when and where it's needed most. Actionable Intelligence Making relevant information instantly accessible, paired with in-the-moment action guidance Instant Expertise Eliminating the learning curve to empower new staff with the confidence and knowledge Elevated Hospitality Freeing your team from cognitive overload to focus on what truly matters: creating genuine, memorable guest connections. Knowing Before Learning Freeing new staff from cognitive overload 
to focus on learning by doing

🧱How we built it

We love XR, its a truly magical tool that gives so many opportunities. The first pain we wanted to solve was the navigation - so we built the mesh nav system that would lead the waiter to the kitchen or to the correct table - indoor navigation style - point-to-point line. Then we, of course, wanted to utilize the heads-up hands-free ability of smart XR glasses - a perfect combination for waiter, as they need to express hospitality and maintain eye-contact with the guests - but with the power of XR they now can get all the relevant and timely information right to their line of sight. We connected an incredibly powerful empathic AI system that can receive large contexts and runs a dynamic model that can run functions (tools) based on different things the user is saying. This allowed us to build a flow that flawlessly leads the newbie through the entire process of greeting and seating the restaurant guests, asking the right questions about dietary restrictions and culinary preferences. And then our newbie as if by magic knows all the daily specials and can even recommend wine pairing options. But the coolest part comes when we got Hume.ai to listen to the guest. This allowed to offload the whole order taking from the newbie waiter - because now the system takes the orders. The guests can jabber away and be picky and creative - Hume.ai can defer the exact dish the restaurant can offer based on the free-speech request and the newbie will be able to read the full proper name of the dish, of course, complimenting the guests for their excellent choice. Our AI is so smart that

♟️Challenges we ran into

Aparently, Quest is an amazing device but its microphone can only listen to the person who wears the quest, and unlike its younger brother - RayBan - Quest cant be used for “translation” or any other conversation case. We had to bring a USB-C dongle with bluetooth microphone to override the Quest mic.

And then just when it was time to test a significant milestone of integration of different pieces of the project - an even that happens less than once a month occurred - Hume service was down. But, they fixed it after 36 minutes, very stressful 36 minutes. And then when it was finally back, the connection was closing right when it was starting. It took us one hour to realize we copied the API key with an extra trailing space. Massive facepalm. But we managed!!!

🏆Accomplishments that we're proud of

Nav mesh with Scene was a brilliant nut to crack and it came out awesome. And the tooling (contextual function calling) with Hume is an absolute win for us.

👨‍🎓What we learned

We realized once again the incredible power of XR, context awareness of AI and the delivery of timely and relevant actionable information. And we learned so much about nav mesh with Scene

🚀What's next for waitAR

Our vision for waitAR extends far beyond this initial prototype. We see a clear, four-stage path forward to revolutionize the service industry.

  1. Immediate Priority: MVP Maturation & Enhanced Contextual AI

Our immediate focus is to refine waitAR into a robust Minimum Viable Product (MVP). A key part of this is deepening our AI-driven contextual awareness. The next major feature will be implementing real-time computer vision for dish recognition. This will empower the system to automatically verify that the correct dishes are being taken to the correct table, drastically reducing order errors and ensuring a flawless guest experience.

  1. Next Step: The Empathic Training Simulator

While waitAR currently excels as an on-the-job "work before you learn" tool, our next development phase will introduce a dedicated AI-powered Training Mode. This offline simulator will fully leverage the empathic AI capabilities of Hume.ai to teach the art of hospitality. New hires will interact with a variety of virtual guest personas in realistic scenarios. Hume.ai's API will analyze the trainee's vocal tone and expression, providing real-time feedback on their perceived empathy, confidence, and warmth. This transforms waitAR into a comprehensive development platform that teaches not just what to say, but how to say it, building true service excellence.

  1. Mid-Term Goal: Expansion into New Hospitality Verticals

Once our platform is mature, we will expand beyond the restaurant floor. The core waitAR engine is designed for versatility, and we see immense potential in:

Hotels: Assisting concierge services with guest information, guiding housekeeping staff, and coordinating event management personnel.

Airlines: Providing cabin crew with real-time passenger information, meal preferences, and connection details.

Retail: Empowering sales associates with live inventory data and customer profiles.

  1. Long-Term Vision: A Platform for Mainstream XR

waitAR was fundamentally built for the next generation of XR hardware. We are keenly aware that the industry is moving towards a future that combines the powerful spatial computing of devices like the Meta Quest with the socially acceptable, all-day form factor of smart glasses. We are ready for this inflection point. As XR glasses become mainstream, waitAR is perfectly positioned to become the essential, ambient software layer for service professionals everywhere, making elite-level service intuitive and accessible to all.

Built With

Share this project:

Updates