The Statue in the Crypt

Inspiration

We were inspired by previous explorations real-time generative AI in games, like Fortnite's Darth Vader, or The Button showcase at Unreal Fest Orlando. We wanted to expand on this as a core gameplay feature

What it does

You play as a treasure hunter seeking treasure. However, a statue housing an all-powerful entity (Google Gemini) stands in your way. You must convince, impress, or deceive the statue to get to the treasure behind the statue.

How we built it

This project was made in Unreal Engine 5, using POST requests to interface with Gemini's API. We used a system similar to a command line to give Gemini influence over the game world. When prompting Gemini, we tell it to put its commands in special characters to distinguish it from dialogue. On the client, we parse Gemini's output and match it with premade commands. These commands support argument passing, similar to a CLI, which can alter the command's behaviors. To give Gemini context of past messages, we also included a message history management system, which is periodically summarized by Gemini to reduce token usage.

Challenges we ran into

Unreal Engine 5 did not have any ready-made Gemini SDKs or API wrappers, unlike other workflows. This required some time extra time to be invested into interfacing with the API. The limitations of the Gemini free-tier were also a big challenge. Only a small number of API requests can be made per minute. Due to the quick back-and-forth conversational nature of the game, this can sometimes cause delays dialogue if the player progresses quickly.

Accomplishments that we're proud of

We were able to make a full gameplay loop with interesting decisions and challenges, using Gemini as a central gameplay feature. We are proud of the power we were able to give Gemini over the game and the player, making it a compelling part of the world.