ASL Quick Draw

Inspiration:

We wanted to make learning American Sign Language fun and accessible. Most ASL learning tools are passive: watch a video, memorize a sign, rinse and repeat. What if you could compete while you learn? The tension of a quick draw duel, the pressure of the countdown, and the rush of getting the sign right in time. We built ASL Quick Draw to prove that accessibility and fun aren't mutually exclusive.

What it does:

ASL Quick Draw is a real-time, Wild West-themed multiplayer dueling game where players compete to perform ASL letter signs faster and more accurately than their opponent. After logging in, players enter a matchmaking queue and are paired based on Elo rating. At "High Noon," both players face off: a random ASL letter appears on screen and each player has 3 seconds to perform the correct sign on camera. Google Gemini's vision AI then judges each performance in real time. First to win 3 rounds takes the duel, gains Elo points, and returns to the queue for the next showdown.

How we built it:

Frontend: React + TypeScript + Vite with Tailwind CSS made the Wild West aesthetic, and WebRTC captured live webcam feeds relayed to Socket.IO for video and game events in real time

Backend: FastAPI + python-socketio running on Render managed game state across concurrent duel rooms

ASL Classification: Google Gemini 2.5 Pro Vision API handled camera snapshots as JPEG bytes with a structured prompt to identify the ASL sign and return a JSON result {matches, detected_sign, confidence}

Matchmaking: A custom Elo-based matchmaker was made with dynamic range expansion (starts at ±150 Elo and widens by 50 every 10 seconds to balance fairness with queue wait times)

Auth & Rankings: Auth0 handled login and stored per-user Elo, wins, and losses in user metadata for cross-session persistence

Challenges we ran into:

• Connecting users for the standoff game using WebSocket
• Deciding which AI model can translate the ASL images most efficiently

Accomplishments that we're proud of:

• A fully playable, real-time competitive game built and deployed in < 24 hours
• Clean, event-driven architecture that handles concurrent multi-room duels without race conditions
• User friendly interfaces that give justice to the Wild West theme
• A lively experience that drew in many onlookers throughout the weekend at Clemson

What we learned:

• Socket.IO relay is a good alternative to full WebRTC for low-latency video in a game context
• Designing a fair real-time synchronization 1 v 1 game is harder than it looks but essential for competitive integrity and keeping users in a fun environment for learning

What's next for ASL Quick Draw:

• Expanding beyond individual letters to full word signs and short phrases
• Creating leaderboards, seasonal resets, and badges for top performers

Built With

• autho
• fastapi
• gemini
• python
• react
• tailwindcss