Checkpoint

Your chessboard plays on anyone's screen

Comment

Inspiration

Electronic chessboards that bridge physical and digital play already exist. They cost $400-$800 and require proprietary hardware. We thought that was absurd. Checkpoint turns any chess set and any phone into a connected board. One player moves pieces on a real board — the camera does the rest. The other plays on a web interface, anywhere in the world. Computer vision running on AWS detects moves in real time and keeps both sides in sync. No sensors, no special hardware, less than a dollar in cloud credits for month long usage.

What it does

One player sets up a physical chess set and points their phone at the board. The other opens a browser. That's it. The phone uses a frame diffing algorithm on a raspberry pi, where Bedrock's vision model analyzes the board state, infers what move was made, and pushes it to the digital player in real time via AppSync subscriptions. Digital moves flow back the other way — the physical player gets a push notification and a TTS callout so they can update the board. Both players stay in sync without touching anything but chess pieces and a mouse.

How we built it

Backend: AWS AppSync handles the GraphQL API and real-time subscriptions. Lambda functions cover game lifecycle, image ingestion, the CV pipeline, and push notifications. DynamoDB stores game sessions and move history. S3 stores board images and triggers the CV pipeline via event notifications. CloudFront serves both web clients. Everything is defined in AWS CDK (Python) and deployed with a single deploy.sh script. Computer Vision: When a board image lands in S3, it triggers a Lambda CV processor that detects the board state, diffs it against the previous state, infers the move, and calls recordPhysicalMove. We use AWS Bedrock (Claude/Nova) for vision-based board analysis and FEN inference — which let us iterate on prompt design in hours instead of standing up a training pipeline. Clients: Physical side ships as both a PWA (vanilla JS) for zero-install camera capture and a React Native/Expo app for push notifications and TTS. Digital side is a vanilla JS web app with an alternative Jac/Jaseci implementation — both backed by the same AppSync API.

Challenges we ran into

AppSync and binary data don't mix. Sending board images through GraphQL mutations was a dead end. We switched to base64 upload direct to Lambda, which writes to S3 — cleaner, faster, and a natural trigger for the CV pipeline. Real-time sync is deceptively hard. Chess has strict turn semantics. Getting AppSync subscriptions right — handling reconnects, deduplicating events, and keeping both clients consistent — required careful subscription design and more edge case handling than expected. Multi-client config management. PWA, Expo, and two web clients all need the same AppSync endpoint and API key. We automated config propagation in deploy.sh so a single deploy keeps everything in sync.

Accomplishments we're proud of

The full loop works: physical player creates a game, digital player joins with a code, moves flow both ways in real time. That sounds simple. It wasn't. Beyond the core loop — single-command deploy across AppSync, Lambda, DynamoDB, S3, and CloudFront; four client implementations against one backend; and a CV pipeline built on Bedrock that can swap in a custom ONNX model later without touching the rest of the stack.

What we learned

AppSync subscriptions are the right primitive for low-latency game state. Pairing them with S3-triggered Lambda keeps the API clean and the CV pipeline decoupled from the real-time layer — each piece does one thing well. Bedrock made the CV pipeline viable in a hackathon timeframe. Training a custom model wasn't on the table in 48 hours. Prompt-based vision inference was — and it was good enough to ship. Supporting both a PWA and a native Expo app forced useful product thinking: the browser is the right default for zero-friction onboarding; native wins for camera UX, push notifications, and TTS.

What's next for Checkpoint

The CV pipeline is the obvious next frontier — hardening Bedrock-based detection across board styles, piece sets, and lighting conditions, with a custom ONNX model in Lambda as the high-volume cost optimization. Beyond that: adaptive capture intervals, better calibration UX, chess clock support, PGN export, and proper game persistence across disconnects.

Built With

Share this project:

Updates