The gameplay with live commentry

RoadCaster

Every Close Call Has a Story.

A 3D driving game where every swerve, dodge, and crash triggers live AI commentary — built from scratch as a computational linguistics problem.

LingHacks VII Submission · Solo Project · Live Demo · GitHub Repo

Inspiration

It started one evening playing FIFA on the PlayStation.

Not even playing seriously — just messing around. But the commentary kept catching my ear. The announcer wasn't reacting to goals. He was tracking moments — a burst of pace, a shoulder drop, a near-tackle. He wasn't reading from a script. He was building a live story around the action as it happened.

And that raised a question that wouldn't go away: has anyone actually built that from scratch?

Not as a feature bolted onto a AAA game with a million-dollar budget. But as a linguistics problem — designed from first principles. Take raw game physics, figure out what's happening, decide how intense it is, and generate speech that sounds like an actual broadcaster calling the play.

Most NLP projects at hackathons are chatbots or sentiment classifiers. They're useful but they don't make you feel anything. The commentary problem felt different — it's messy, contextual, emotional, and deeply tied to how humans actually use language in real-time performance situations.

That's how RoadCaster was born.

The Core Idea

Most games play background music. RoadCaster plays live commentary.

The challenge isn't generating text. The challenge is making it feel real:

Contextual — the commentator knows what's been happening for the last 18 seconds, not just this frame
Non-repetitive — never says the same thing twice in two minutes
Emotionally matched — the energy of the words matches the intensity of the moment
Strategically silent — knows when to shut up, because real broadcasters use silence for tension

All of that comes together in the Context-Aware Sports Commentary Engine (CASCE) — the NLP backend that powers everything.

How It Works

The Pipeline

The whole system flows through six stages, each with a distinct job:

$$ \text{Game Physics} \;\longrightarrow\; \text{Event Queue} \;\longrightarrow\; \text{Narrative Engine} \;\longrightarrow\; \text{Commentary Engine} \;\longrightarrow\; \text{TTS} \;\longrightarrow\; \text{Voice Output} $$

Stage 1 — Capturing What's Happening

Every frame, the Three.js 3D engine checks the physical relationship between the player's car and every traffic vehicle on the highway using bounding-box collision detection.

From those physics checks, the game captures atomic events: near misses, overtakes, lane swerves, squeeze-throughs, speed boosts, and crashes. These get queued up and sent to the Python backend every 3.5 seconds as a batch.

Crashes and game starts are special — they skip the queue entirely and fire immediately. Because if someone just slammed into a truck, the commentator better react now, not in 3.5 seconds.

Stage 2 — The Narrative Engine (The Brain)

This is where the real linguistics happens.

The NarrativeEngine doesn't look at one event in isolation. It maintains a rolling 18-second window of everything that's happened recently:

$$ \mathcal{W}(t) = { e \mid t - e_{\text{time}} \leq 18\text{s} } $$

Why 18 seconds? Because that's roughly how long a "passage of play" lasts in real sports. Frame-by-frame is too noisy — you'd be screaming about nothing. Looking at the whole game is too slow — you'd miss the moment entirely. 18 seconds is the sweet spot where the commentary can feel both reactive and contextual.

From that window, the engine computes stats: how many overtakes, how many near misses, average speed, how long since the last dangerous moment, the current combo streak. Then it feeds all of that into a priority-ordered rule engine that maps the situation to one of 20 narrative states:

Priority	State	What It Means
1	Crash	Collision — always top priority
2	Extreme Near Miss	Centimetres from death
3	High Combo	10+ consecutive near-misses
4	Clutch Survival	3+ near-misses in the window — barely alive
5	Panic Driving	Lots of events, slow speed — struggling
6	Traffic Chaos	6+ events at high speed — pure mayhem
7	High Risk Driving	Near misses at 140+ km/h
8	Elite Reactions	Combo streak ≥ 5
9	Untouchable	15s clean + 4 overtakes
10	Dominant Run	150+ km/h average, 5+ overtakes
11	Highway Wizard	Constant lane changes + constant overtakes
12	Traffic Surgeon	Clean overtaking with zero near-misses
13	Precision Driving	Long clean streak with strategic lane changes
14	Total Control	30+ seconds of flawless driving
15	Masterclass	25+ seconds clean
16	Flow State	20s clean with active overtaking
17	Controlled Aggression	Aggressive but clean
18	Record Pace	Speed above 175 km/h
19	Building Momentum	Getting into a rhythm
20	Traffic Weaving	Active lane switching

The ordering matters. A crash always takes priority over everything. An extreme near miss always beats a high combo. This reflects what a real commentator would instinctively react to first — the most dramatic thing happening right now.

Transitions — Narrating the Arc

Here's where it gets interesting. Real commentators don't just describe the current state — they talk about how things are changing.

"He's finding his stride now!" isn't about what's happening. It's about the shift from chaos to control.

The engine tracks state transitions: when the narrative state flips (say, from Building Momentum → Traffic Surgeon), it records what changed and how long the previous state lasted. The CommentaryEngine has templates specifically for these transitions, so the commentary narrates the story arc of the drive — not just disconnected snapshots.

Strategic Silence

Not every tick should produce speech. If the driver has been cruising in Total Control for three consecutive ticks, the commentator goes quiet. Silence builds tension. The next line, when it finally drops, feels earned.

This is controlled by a should_speak flag — crashes and extreme near-misses always force speech, but calm repeating states get suppressed. Getting this right was one of the hardest parts of the whole project.

Stage 3 — The Action Narrator (FIFA-Style Play-by-Play)

This is the part directly inspired by FIFA.

On top of the big-picture narrative states, a second engine — the ActionNarrator — works with the raw physical actions captured in each tick: swerve_left, dodge_right, squeeze_through, boost_start, speed_surge.

The key insight: listing actions individually sounds like a robot reading a log file. Grouping them semantically sounds like a broadcaster.

So the narrator groups related actions together:

Raw Actions	Narrated As
`overtake × 3` in 4 seconds	"Carves through the pack, passing 3 cars!"
`squeeze_through` + `dodge_left`	"Threads the needle — slips past the truck!"
`swerve_left` + `swerve_right` + `swerve_left`	"Zigzagging through traffic!"
`boost_start` + `speed_surge(187)`	"Hits the boost — redlines the engine at 187 km/h!"

Then it applies excitement-level formatting:

Calm → lowercase connectors, simple periods. "Passes on the left and accelerates."
Excited → mixed case, exclamation marks, dashes. "Carves through the pack — hits the boost!"
Hype → ALL CAPS, double exclamation, dramatic connectors. "DODGES THE TRUCK AND STILL GOING — THREADS THE NEEDLE!!"

Finally, short situational closers are appended about 70% of the time — little 3-5 word phrases tied to the current state:

Traffic Surgeon → "Clinical precision!" Clutch Survival → "How is he still alive?!" High Combo → "UNSTOPPABLE!!"

These closers are what push the output from sounding generated to sounding broadcast.

Stage 4 — Anti-Repetition

Nobody wants to hear the same line twice in two minutes. That kills the illusion instantly.

The solution is a sliding memory of the last 10 spoken lines. Before selecting a template, the engine filters out anything that's been said recently:

$$ \text{EligiblePool} = \text{AllTemplates}_{\,state} \;\setminus\; \text{Last10Spoken} $$

If the entire pool has been used (small pool, long session), it falls back to the full set rather than going silent. This keeps sessions feeling fresh even after 15-20 minutes of play.

Stage 5 — Three Commentator Personalities

Every narrative state has its own template library for each of three modes:

Mode	Voice	Personality	On Crash
Sports Broadcast	Jasper	Hyped, excited, exclamatory	"AAAAAAH! HA HA HA! SPECTACULAR DESTRUCTION!"
Dramatic Narrator	Hugo	Cinematic, dark, trailing off	"A flash of chrome, a screech of tires, and then... ha ha ha... silence."
Savage Critic	Kiki	Sarcastic, mocking, savage	"HA HA HA! Wrecked! Hope you have insurance, because that was pathetic!"

The Savage mode was genuinely interesting to design. Insult comedy has its own timing — if the mockery fires too early before anything interesting happens, it falls flat. Because Savage uses the same state-awareness system as the other modes, it only escalates the roasting when there's actually something worth roasting.

Stage 6 — Voice (KittenTTS)

The final commentary text gets synthesised into actual spoken audio using KittenTTS Nano — a 15-million-parameter neural text-to-speech model that runs entirely locally. No API calls, no cloud dependency, no latency from external services.

$$ \text{Commentary Text} \;\xrightarrow{\;\text{KittenTTS}\;}\; \text{24kHz WAV} \;\xrightarrow{\;\text{Web Audio API}\;}\; \text{Speakers} $$

The audio pipeline has a built-in lock (isSpeaking) that prevents overlapping speech. Once the commentator starts talking, no new requests fire until the line finishes. Getting this lock right was tricky — more on that in Challenges.

Challenges

The Silence Problem

Early versions were way too chatty. A new line every 3.5 seconds, no matter what. It sounded exhausting, not exciting. Real broadcasters use silence to build tension — when the driver is threading through dense traffic in total focus, the audience holds its breath.

Building should_speak suppression to handle this properly took multiple iterations. The rule: suppress consecutive calm states, but never suppress crashes or extreme near-misses. The result is a commentary rhythm that mirrors the emotional rhythm of the drive.

Sentence Length vs. Game Speed

There's a real tension between making commentary sound good and keeping it timely. A beautifully crafted 8-second sentence is stale by the time it finishes — the game has moved on. Hard lesson learned: short punchy phrases always win in real-time commentary.

Final rule: 20-word maximum on all ActionNarrator output. No exceptions.

The Audio Race Condition

The most frustrating bug: the isSpeaking flag was being set to true after the TTS network request returned — not when it started. In that tiny async gap, a new narrative tick would check isSpeaking, see false, and fire a second TTS request. Result: two commentary lines playing on top of each other.

The fix was embarrassingly simple: move isSpeaking = true to the first line of the function, before the fetch() call. One line of code. Took hours to figure out.

Keeping It Fresh

Even with anti-repetition, a pool of 5 templates gets stale after 3 minutes. Two solutions: expand every pool to at least 4-6 templates, and lean on the ActionNarrator as a secondary source — since its output is combinatorially generated from vehicle types, directions, counts, and speeds, it produces way more variety than a fixed template library ever could.

What This Taught Me

Going into this, it felt like a frontend challenge: make a fun game, slap on some voice. Coming out of it, it's clear that it was almost entirely a linguistics challenge.

Getting commentary to feel right meant thinking about sports broadcasting as a communicative act. A broadcaster does four things simultaneously:

What They Do	How RoadCaster Does It
Describe what just happened	`ActionNarrator` — play-by-play from raw actions
Contextualise against recent history	`CommentaryEngine` — state-level ambient commentary
Build emotional stakes	Excitement levels, transition arcs, momentum language
Manage attention	`should_speak` suppression, silence as punctuation

None of those are typical engineering problems. All of them are linguistics problems dressed up in code. That's the real takeaway from this whole project — and honestly, it's what made it so much fun to build.

Tech Stack

Layer	Technology
3D Engine	Three.js 0.184.0, WebGL
Frontend	Vanilla JavaScript (ES Modules), HTML5, CSS3
Backend	Python 3.12, Flask 3.0
NLG Engine	Custom `NarrativeEngine` + `ActionNarrator`
TTS	KittenTTS Nano (15M params, ONNX)
Deployment	Hugging Face Spaces (Docker)