GemTrails

Inspiration

The pandemic was a pivotal moment in history. It drove many businesses to drastically adapt to survive. One of the hardest hit were the brick-and-mortar businesses in the area, businesses like restaurants and cafes. This moment of realisation inspired us to look for a solution. For all the trouble the pandemic caused, it did bring about a digital revolution. Many businesses that adapted digital solutions to keep the business running (e.g. QR codes, delivery, apps, etc.) have kept these programmes running in the present. Despite all this, roughly 70% of customers never make a return visit. AR technology has proven it has the potential to significantly increase brand recognition and customer engagement. With brick-and-mortar businesses adopting digital solutions and with AR technology growing more mature, we thought the time was right to bring an AR solution to tackle the problem.

What it does

GemTrails is aiming to be an immersive AR treasure hunt experience that rewards players with discounts while offering businesses marketing and brand awareness opportunities through gameplay. For the purpose of the hackathon, we built a small proof of concept that can be expanded to encompass the vision.

How we built it

The app uses a combination of different technologies to realise the vision: Unity: The game engine ARFoundation: Allows us to work with ARCore/AR Kit features such as plane tracking ARCore: Allows us to build the AR app for Android and iOS Google’s Geospatial API: Extends ARCore to give advanced geolocation capabilities Geospatial Creator: Helps us develop experiences from the comfort of our home. Keeps us from making silly mistakes before demo day (like not accounting for the pedestrian underpass) Google’s Directions API: Allows us to dynamically determine personalised routes for players Protobuf: Allows us to serialise the treasure hunt experiences for eventual transmission over the internet

How it works:

A player is presented with a series of challenges They need to complete the challenge in a sequence to unlock the reward at the end For the proof of concept, we used 1 puzzle minigame and a short follow-the-trail challenge.

Challenges we ran into

Getting accurate geospatial information is still difficult

Out on the field, it is really frustrating when the anchors are off. In areas where there are tall buildings, geolocation information is still unreliable despite using a VPS. We’ve had to compromise and build the prototype using a hybrid of AR technologies rather than relying exclusively on the VPS.

Geospatial Creator consumes a lot of API requests by default

This wasn’t a problem when photorealistic tiles weren’t GA. But once it went GA, the costs shot up really quickly. We were lucky to have cloud credits to spare, but this would be an unwelcome surprise if we didn’t pay attention.

ARCore Semantics API didn’t work

We had trouble enabling the ARCore Semantics API on our testing devices (iOS) because of package conflicts. Seems like the iOS isn’t quite stable yet. Accomplishments that we're proud of Developing a geolocation-inspired experience is difficult, because it is really hard to get a tight feedback loop without actually being in the location. This can really impede the creative process. Through our experiences with the Geospatial Creator, it has enabled us to establish a workflow that kept the creative juices flowing while helping us make progress with the application. We didn’t get to the point we wanted to (there are some nasty hacks in there), but overall we made good progress and the proof-of-concept definitely did it’s job.

What we learned

Geolocated AR applications are still hard to develop. No amount of preparation can prepare you for conditions on the day (weather, lighting, public transport disruptions, etc.). This also means we have no way to predict how our players will experience the situation either. However, the few times we get it right, we were well rewarded for our troubles. We do believe in a future where AR applications will seamlessly integrate into our daily lives, but we are still far from it.

What's next for GemTrails

We have a few more ideas we want to try: More games – Minigames like solving puzzles, whac-a-mole, and tetris never made it to the proof of concept. We’d like to add these options to give more variety to players Gen AI – Buzzword of the decade. Using a combination of Unity Sentis and LLMs like OpenAI (through whisper), we can add more personalisation into the app that goes beyond making routes ARCore semantics API – We never got it to work this time, but this will allow us to create more intricate puzzles that can be deployed at any location (like find the nearest tree).