K.A.R.E.N

Main "voice interaction" profile
Low-Latency Breakout Game Option
Optimized Power-Saving Screen. ROSS Style!
Original blueprints for KAREN as a hardware-based assistant, potentially making use of a BracketBot (maybe for another time!)

Inspiration

The idea originally sprouted from a more robotics-oriented project one of our team members originally brainstormed during Hack the North. When the ROSS FPGA challenge dropped, the plan very quickly arose that it was the perfect way to use this idea towards building a full voice-enabled interface on real low-level hardware (graphics, input, audio, and “personality”) while still being able to ship something fun in a weekend. So K.A.R.E.N was born: part assistant, part toy, part screensaver… and yet 100% committed to being unhelpful. Ragebait her enough and you'll suffer her wrath.

What it does

Gemini-powered interface to “talk” to K.A.R.E.N: you type or push-to-talk, and an AI decides what FPGA-side actions to trigger (nagging, mode switches, prompts, etc.).
Keyboard input to control everything: play the game, interact with the assistant, and manually trigger the screensaver via signals to the FPGA.
Breakout, played with native keyboard input (no emulator - real signals driving real logic).
ROSS Video screensaver that kicks in when you leave K.A.R.E.N unattended (of course we need screen protection + it looks cool)

How we built it

FPGA (SystemVerilog) core

Implemented part of the video pipeline (HDMI timing / color output) and a simple UI state machine to swap between assistant, game, and screensaver modes.
Built Breakout as a combiantion of both software and hardware logic: ball physics, paddle control, collision checks, score/state, plus sprite/shape rendering. Wrote a Python controller that:
Captures keyboard events + push-to-talk,
Calls Google Gemini to interpret what the user wants (or what K.A.R.E.N thinks they want), while responding with an appropriate "mood"
Translates the AI decision into commands/signals that the FPGA understands.
Used a “command vocabulary” (mode changes, triggers, parameters) so the FPGA stays deterministic and fast while the AI stays flexible.
Added audio output for K.A.R.E.N’s voice cues using Elevenlabs, while using a local-Python-based STT model for audio input. The on-screen waveform adjusts as packets are sent in. Tooling / integration
Used FPGA vendor tooling (AMD Vivado/Tcl + debug cores like VIO/logic probing) to iterate quickly and verify timing while adding features.

Challenges we ran into

Generating the waveform

With limited compute on the device, calculating sine positions in real time was expensive. We switched to a lookup table approach to reduce overhead while keeping the signal display stable and clean. Timing + real-time constraints
Video output is unforgiving: if your timing is off, the display tells you immediately. Getting stable sync + clean rendering while adding game logic took careful iteration. Compilation + environment issues
Mid-build (around 2AM), we had to swap machines/hardware, reconfigure the toolchain, and get everything compiling again without losing momentum. Bridging “AI decisions” to “hardware truth”
The AI can be chaotic; the FPGA cannot. Designing a command interface that was expressive and safe/deterministic took more thought than we expected.

Accomplishments that we're proud of

The waveform (and getting audio to behave on low-level hardware)
The Google Gemini integration driving actual hardware behaviors
Shipping a full interactive experience on an FPGA (for some of us, our first time going this low-level)
Breakout skills improved dramatically (unfortunately)
The voice matches our reference: K.A.R.E.N really does sound like the kind of computer that's fun to ragebait

What we learned

How FPGAs “think”: state machines, timing, constraints, debugging. synthesising (not compiling!)
How to trade compute for memory using lookup tables
How to design a clean interface between high-level AI and deterministic hardware
How to iterate fast with hardware using debug tools instead of vibe-coding our way through (no React-based Web3 B2B blockchain SAAS here)

What's next for K.A.R.E.N - the world's least helpful FPGA-powered assistant

FREE KAREN FROM HER METALLIC PRISON. give her wheels, give her access to the internet, give her more games. possibilities are endless from here. Maybe a full-blown operating system.
Make her nag you better - more contextual commentary, personalities/modes, and better timing so it interrupts you optimally.
Improve AV quality: smoother animations, richer audio synthesis, and better asset storage (ROM/SD/streamed frames).