Send.

Inspiration

Climbing at a new gym or outdoor wall can be overwhelming. There are dozens of holds, but no clear path forward. We wanted to democratize route-setting expertise and make climbing more accessible. What if anyone could snap a photo and instantly see an optimal path up the wall? That's Send: your smart bouldering buddy that sees what the pros see.

What it does

Send is a climbing app where you snap a photo of any wall and instantly see optimal routes mapped out using computer vision.

Key features:

Instant Route Detection: Point your phone at any climbing wall, take a photo, and our YOLO-powered backend detects all the holds and draws a suggested bottom-to-top route
Session Tracking: Log today's climbs, sends, and time on the wall, building your climbing portfolio automatically
ArUco Calibration: Uses printed ArUco markers to convert pixel distances into real-world centimeters for accurate route planning

How we built it

Backend (Python + FastAPI + OpenCV):

YOLO (Ultralytics) for real-time hold detection with 85%+ confidence threshold
OpenCV for image processing, ArUco marker detection, and route overlay rendering
Custom route planner algorithm that picks holds bottom-to-top using pixel-space heuristics, balancing vertical gain with horizontal movement
FastAPI serving the REST API with multipart image upload

iOS App (SwiftUI):

Native SwiftUI interface with camera integration
Real-time photo capture → API upload → route overlay display flow
Google OAuth for user authentication

Stack: Python / FastAPI / PyTorch / YOLO / OpenCV / Swift / MongoDB

Challenges we ran into

Real-World Scale Calibration: Converting pixel distances to centimeters using ArUco markers—handling cases where markers are partially visible or at image boundaries
iOS ↔ Backend Communication: Getting the camera capture → API upload → route display pipeline smooth, especially handling network errors gracefully on physical devices vs. simulators
Route Planning Heuristics: Balancing between "climbable" routes (reasonable reach distances) and "interesting" routes (not just straight up the middle)

Accomplishments that we're proud of

End-to-end working prototype: From camera → YOLO detection → route planning → annotated PNG overlay, all in under 2 seconds
Native iOS experience: Smooth camera integration with grid overlay, live preview, and seamless navigation to route analysis
Clean architecture: Modular separation between detection, route planning, and rendering; easy to swap algorithms or add new features
Production-ready API: FastAPI with health checks, file validation, proper error handling, and Google OAuth

What we learned

How to deploy YOLO models for custom object detection (climbing holds aren't in ImageNet!)
ArUco markers are surprisingly powerful for real-world scale calibration in CV applications
The importance of tunable parameters. Our route planner has 6+ configurable thresholds that dramatically change output quality
Testing CV pipelines end-to-end requires real images; unit tests catch regressions but can't validate "does this look like a climbable route?"

What's next for Send

Smarter Route Planning: Use climber biomechanics (arm span, height) to generate personalized routes, not just geometric paths
Video Analysis: Record climbs and overlay real-time pose estimation to analyze body positioning and suggest technique improvements
Outdoor Wall Support: Extend hold detection to natural rock features, not just gym holds
Social Features: Share routes, compare beta with friends, and crowdsource route ratings
Apple Watch Integration: Heart rate tracking and quick-log sends from your wrist mid-session