Caltech x Ironsite
Spatial Intelligence in the Physical World
April 24th – April 26th
Can you teach a machine to truly understand the physical world?
Today’s AI can see, but doesn’t comprehend. It recognizes objects but misses the critical spatial relationship that humans understand intuitively.
Ironsite is hosting a 36-hour hackathon to solve this problem. We’re looking for builders that can bridge the gap between seeing and understanding.
For the first time, we’re opening up our data. Participants will have access to hours of real construction footage—workers building, tools in motion, the physical world unfolding in real time. No synthetic datasets. No toy problems. Just raw, messy reality. What you build with it is up to you.
Vision Data Captured Directly From Job Sites
–
Your Mission If You Choose to Accept It:
Define a Problem:
Pinpoint a key spatial task where current models fail. Find a visual task that is trivial for humans, but AI models completely fail at. Use existing models like Gemini 2.5 Pro, Claude Opus, or GPT5 to give them an image or video and ask them a question that reveals their lack of spatial intelligence. We will provide API credits for you to use.
Develop a Technique
Create an innovative solution to solve the problem. Utilize prompt engineering, simple fine-tuning, inference time compute strategies, or any other clever techniques you can think of to improve how well the models can perform on that task. Be detailed and scientifically minded.
Showcase the Impact:
Demo your technique on a real-world problem. Your idea doesn’t have to solve the problem perfectly, but rather show how these models can be augmented to increase their spatial intelligence, even for a narrow use case or task. It’s even better if the task you solve could have real world impact.
The Details
—
Prize Pool:
5,000 first place, 2,000 second, 1,500 third
Timeline:
April 24th – April 26th
36 Hours of Innovation
Teams:
Teams of 1-4.
Why Apply:
This is your chance to work on a frontier problem, get noticed, and win big.
Meet the Ironsite Team
Keenan Brekke – FDE
Former Superintendent of Pacific Structures and has led 100+ crews across SF-Bay Area projects.
Daniele More – CSO
Former research tech lead at Google DeepMind and head of model research at the AI-HW startup Etched.
Charu Thomas – CTO
Founder and CEO of OX ($20m raised), an AI-powered wearable for frontline workers.
Max Mona – Founder
A second-generation construction builder and South Park Commons Founder Fellow.
A