Upload hardhat camera footage → Get productivity insights via chat. That's it.
Built in 48 hours for UMD x Ironsite Spatial Intelligence Hackathon (Feb 20-22, 2025)
Watch the demo: SiteIQ Demo Video
Construction supervisors watch hours of hardhat footage but can't answer:
- "Was the crew productive today?"
- "How much time was wasted searching for tools?"
- "What was the productivity during the critical 2-hour window?"
Current AI (ChatGPT, Claude) can describe what they see but can't quantify productivity over time.
SiteIQ analyzes egocentric construction video and answers those questions in plain English.
Input: Construction worker POV video (MP4) Output: Productivity score, insights, natural language Q&A
# Try it yourself (5 minutes)
git clone https://github.com/khetansarvesh/spatial_intelligence_ironsite_hackathon.git
cd spatial_intelligence_ironsite_hackathon
pip install -r requirements.txt
python main.py --video demo_video.mp4 --max-frames 300
# Start dashboard
cd dashboard && npm install && npm start
# Open http://localhost:3000 → Upload video → Ask questionsAutomated Analysis Output:
✅ Productivity Score: 95.6% (Exceptional)
✅ Active Time: 12.7s (95.5%)
✅ Idle Time: 0.0s (0.0%)
✅ Dominant Activity: Precision block alignment
⚠️ Insight: 17 short work segments detected
💡 Recommendation: Reduce interruptions for longer continuous workflows
Supervisor asks via chat: "What was the worker doing most?" SiteIQ responds: "Precision work on block alignment - 95.5% of the time. Exceptional focus maintained throughout."
Works in real-world conditions:
- ✅ Construction gloves (thick leather)
- ✅ Variable lighting (indoor/outdoor)
- ✅ Camera motion (worker moving)
- ✅ Cluttered job sites
- ✅ Multiple trades (masonry, framing, electrical, plumbing)
┌─────────────────────────────────────────────────────────────────────┐
│ SiteIQ Pipeline │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Video │───▶│ Perception │───▶│ Temporal │ │
│ │ Input │ │ Pipeline │ │ Analysis │ │
│ └──────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────────────────┐ │
│ │ Frame Information JSON │ │
│ │ (HOI data for each frame) │ │
│ └─────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────┼───────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Summary │ │ CodeAct │ │ Evidence │ │
│ │ Agent │ │ Agent │ │ Agent │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Markdown │ │ Answer │ │ Video │ │
│ │ Summary │ │ + Code │ │ Clips │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ Web Dashboard │
│ (Chat Interface + Video) │
└─────────────────────────────┘
| Component | Description | Technology |
|---|---|---|
| Perception Pipeline | Hand detection, tool detection, HOI analysis | MediaPipe, GroundingDINO, YOLOv8 |
| Temporal Analysis | Activity classification, productivity scoring | State machine, temporal segmentation |
| Summary Agent | Generates markdown productivity reports | Claude API |
| CodeAct Agent | Answers questions by generating & executing Python code | DSPy, Claude API |
| Evidence Agent | Finds relevant video timestamps, clips evidence | Claude API |
| Web Dashboard | ChatGPT-style chat interface with video playback | Node.js, Express, Vanilla JS |
Video (30 FPS)
↓
[1] PERCEPTION - What's happening right now?
→ Hands detected? (MediaPipe)
→ Tools in use? (YOLO - drill, hammer, saw, etc.)
→ How are hands moving? (Optical flow)
↓
[2] TEMPORAL ANALYSIS - What activity is this?
→ Activity classifier: 7 states (active tool use, precision work,
material handling, setup, searching, traveling, idle)
→ Each state has productivity weight (0% to 100%)
↓
[3] SESSION INTELLIGENCE - Overall patterns?
→ Productivity score (weighted time average)
→ Idle periods, tool switches, peak performance
→ Auto-generated insights & recommendations
↓
[4] CONVERSATIONAL INTERFACE - Ask questions
→ CodeAct agent generates Python code to query data
→ Evidence agent finds video timestamps for proof
→ Natural language: "Was productivity better in morning?"
Key Innovation: We combine what's visible (hands, tools) with how it's moving (motion patterns) to classify construction-specific activities over time. The CodeAct agent writes executable Python code to answer questions, providing transparency and accuracy.
| Feature | SiteIQ | Generic AI (ChatGPT/Claude) | Traditional Time-Motion Study |
|---|---|---|---|
| Understands time/productivity | ✅ Yes | ❌ Frame-level only | ✅ Yes |
| Construction-specific | ✅ 7 activity states | ❌ Generic descriptions | ✅ Manual observation |
| No code needed | ✅ Chat interface | ❌ API/technical | ✅ Pen & paper |
| Shows generated code | ✅ Transparent reasoning | ❌ Black box | ❌ N/A |
| Video evidence clips | ✅ Auto-clips proof | ❌ No | ❌ Manual |
| Automated | ✅ Fully | ❌ Manual labor |
Bottom line: First system that combines computer vision + temporal analysis + code-generating AI specifically for construction productivity.
- Hands alone: 62% activity accuracy
- Tools alone: 58% accuracy
- Motion alone: 71% accuracy
- All combined: 83% accuracy ← 21 percentage point improvement
- Correlation coefficient: r = 0.78 between hand visibility and productive work
- When hands disappear: Usually searching (panning camera) or idle
- Agent generates Python code, executes it, returns answer
- User can toggle to see exact code that computed the answer
- No hallucination - grounded in actual data queries
- Evidence agent identifies timestamps supporting each answer
- Dashboard clips ±1 second around each timestamp
- Supervisors can verify AI claims with video proof
- Raw frame-by-frame: 40 state transitions/minute (noisy)
- 3-frame sliding window: 8 transitions/minute (realistic)
spatial_intelligence_ironsite_hackathon/
├── src/
│ ├── perception/ # Computer vision components
│ │ ├── hand_detector.py # MediaPipe hand tracking
│ │ ├── tool_detector.py # GroundingDINO/YOLO tool detection
│ │ └── hoi_detector.py # Hand-object interaction logic
│ ├── temporal/ # Time-series analysis
│ │ ├── activity_classifier.py
│ │ └── session_aggregator.py
│ └── agent/ # LLM agents
│ ├── agent.py # CodeAct agent (generates Python)
│ ├── evidence.py # Evidence extraction
│ ├── summary.py # Report summarization
│ ├── tools.py # Agent tool functions
│ └── prompts.py # System prompts
├── dashboard/ # Web interface
│ ├── server.js # Express backend
│ └── public/
│ ├── index.html
│ ├── style.css
│ └── script.js
├── outputs/ # Generated files
│ ├── frames_information.json
│ ├── final_report.json
│ ├── productivity_summary.md
│ └── annotated_video.mp4
├── main.py # Video processing pipeline
└── requirements.txt
git clone https://github.com/khetansarvesh/spatial_intelligence_ironsite_hackathon.git
cd spatial_intelligence_ironsite_hackathon
# Install dependencies
pip install -r requirements.txt
cd dashboard && npm install
# Set API key
export ANTHROPIC_API_KEY=your-key-here
# Start dashboard
npm start
# Open http://localhost:3000
# Upload video → Chat with AI# Process video
python main.py --video your_video.mp4 --max-frames 300
# Query results
python query_agent.py --report your_video_report.json --summary| Endpoint | Method | Description |
|---|---|---|
/api/summary |
GET | Get markdown productivity summary |
/api/video/annotated |
GET | Serve annotated video |
/api/ask |
POST | Ask a question (returns answer + generated code) |
/api/evidence |
POST | Get video timestamps for evidence |
/api/video/clip |
GET | Get clipped video segment (±1 sec) |
/api/health |
GET | Health check |
┌─────────────────────────────────────┐
│ SiteIQ │
├─────────────────────────────────────┤
│ │
│ You: [video thumbnail] │
│ Analyze this video │
│ │
│ Agent: ✓ Analysis complete │
│ [Annotated video player] │
│ │
│ Session: 13.3s masonry work │
│ Productivity: 95.6% (Exceptional) │
│ │
│ You: What was productivity 5-10s? │
│ │
│ Agent: [Video clip evidence] │
│ Productivity was 100% between │
│ 5-10 seconds. [Code toggle] │
│ │
├─────────────────────────────────────┤
│ 📎 Ask follow-up... ➤ │
└─────────────────────────────────────┘
Try asking the dashboard:
- "What was the overall productivity score?"
- "How much idle time was there?"
- "What tools were used?"
- "When was peak productivity?"
- "What activity took the most time?"
- "Show me the productivity between 5s and 10s"
- Computer Vision: MediaPipe, GroundingDINO, YOLOv8, OpenCV
- LLM Framework: DSPy, Anthropic Claude
- Backend: Node.js, Express
- Frontend: Vanilla JavaScript, highlight.js (syntax highlighting)
- Video Processing: OpenCV, FFmpeg
Detection Accuracy (validated on 100 frames):
- Hand Detection: 94% precision, 89% recall
- Tool Detection (YOLO): 78% precision, 72% recall
- Activity Classification: 83% agreement with human labelers
Processing Speed (MacBook Pro M1):
- YOLO + GPU: 8-10 FPS (real-time factor: 0.3x)
- YOLO + CPU: 3-5 FPS (real-time factor: 0.15x)
Practical: 1 minute of video → 10-30 seconds processing time
UMD x Ironsite Spatial Intelligence Hackathon (Feb 20-22, 2025)
| Person | Role | Contribution |
|---|---|---|
| P1 | Perception Lead | Hand detection (MediaPipe), HOI integration |
| P2 | Perception | Tool detection (YOLO/DINO), Scene classification |
| P3 | Temporal Lead | Motion analysis, Activity FSM, Session aggregator |
| P4 | Agent Lead | LLM integration, CodeAct agent, Evidence agent |
| P5 | Integration Lead | Pipeline, Dashboard, Testing, Documentation |
Construction productivity hasn't improved in 40 years while other industries transformed with AI.
The problem: Existing AI can describe but not quantify. Construction supervisors need numbers, not narratives.
Our solution: First end-to-end system that converts egocentric video → productivity metrics → natural language insights with video evidence.
This isn't just a hackathon project. This is the foundation for AI-powered workforce analytics in construction.
MIT License
Built with passion in 48 hours. Ready for production.