Gambit

The start of a poker hand. Bounding boxes surround important clickable segments of the game.
The end of a poker hand. Bounding box creation is dynamic and is able to handle sliders and mouse drags.
Little Alchemy game. Bounding boxes around all dragable elements with proper click, drag, and drop function.

Inspiration

Gambit addresses a critical gap in the $300 billion global video game market by introducing a voice-enabled AI to enhance online game accessibility. Our solution serves the massive, yet underserved, population of gamers with disabilities.

What it does

Gambit listens to a user's voice and then executes the in-game actions, such as moving a character or initiating a sequence of moves, all in real time. By converting voice commands into complex inputs and information, Gambit is positioned not only as a crucial social good but also as a powerful economic driver that unlocks a significant segment of the market.

How we built it

Together, these components form a seamless voice-to-action system adaptable to nearly any game or platform.

🧠 Google Gemini – Bridges the gap between computer vision and spoken commands. Converts spoken instructions into intelligent, context-aware action sequences.
🗣️ Deepgram – Handles accurate, low-latency speech-to-text conversion, with the ability to conversational cues and pauses.
👁️ Computer Vision – Allows Gambit to “see” and interpret on-screen environments dynamically.
🐍 Python Automation Engine – Executes the translated commands by controlling keyboard and mouse inputs directly.

Challenges we ran into

We ran into challenges with Omniparser, the technology we used for image segmentation. Downloading the process was smooth on MacOS, but took multiple hours for Windows.

Accomplishments that we're proud of

We’re most proud of Gambit’s adaptability. It can interpret and respond to any visual interface without needing game-specific integrations — a major step toward universal gaming accessibility.

What we learned

We explored new domains across AI vision, AI speech to text technologies, and real-time system control. More importantly, we learned how these technologies can intersect to create meaningful, inclusive innovation that extends beyond entertainment.

What's next for Gambit

Next, we plan to:

Expand Gambit’s compatibility to include a wider range of game genres (e.g., FPS, strategy, racing).
Integrate customizable command mappings for different player needs.
Partner with game studios and accessibility organizations to scale our impact.
Explore mobile and VR adaptations, making hands-free play accessible across devices.