Real-time hybrid AR object detection - Fast local YOLO + Accurate Gemini refinement!

┌─────────────────────────────────────────────┐
│  COCO-SSD (Local)    →  Instant AR Overlays │
│  ⚡ 10-30 FPS            ✅ Always visible   │
└─────────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────────┐
│  Gemini (Backend)    →  Label Refinement    │
│  🧠 0.5 FPS              ✨ Better accuracy │
└─────────────────────────────────────────────┘

Best of both worlds:

• ⚡ Fast: COCO-SSD gives instant visual feedback
• 🎯 Accurate: Gemini refines labels in background
• 🎨 AR: Labels stick smoothly to objects in 3D space

• ✅ Instant Detection - COCO-SSD shows AR overlays immediately
• ✨ Smart Refinement - Gemini upgrades labels for accuracy
• 🎯 Advanced AR Tracking - Labels stick to objects (Google Lens style)
• 📱 Mobile Optimized - Works on iOS and Android
• 🔊 Voice Feedback - Audio alerts for important signs
• 📹 Dashcam Mode - Record video with detections
• 🌐 PWA - Install as native app
• 🚶 Fall Detection - Emergency pause and recording

git clone https://github.com/YOUR_USERNAME/SignVision-AR.git
cd SignVision-AR

# Install Python dependencies
pip install -r requirements.txt

# Set up Gemini API key
cp .env.example .env
# Edit .env and add your GEMINI_API_KEY

1. Go to Google AI Studio
2. Click "Create API Key"
3. Copy and paste into .env file

# Start backend (Gemini refinement)
python server.py

# In a new terminal, serve frontend
python -m http.server 8080

# Open http://localhost:8080

1. Camera captures frame (1920x1080)
2. COCO-SSD detects objects (50-150ms)
   • Shows AR overlays immediately
   • Labels: Traffic lights, stop signs, vehicles, pedestrians
3. Gemini refines labels (background, every 2 seconds)
   • More accurate classification
   • Detects walk/no walk signals
   • Identifies specific sign types
   • Upgrades COCO labels with ✨ sparkle
4. AR tracking keeps labels stuck (Google Lens style)
   • IoU matching
   • Motion prediction
   • Camera motion compensation
   • Exponential smoothing

• Regular box (3px): COCO-SSD detection
• Thick box (4px) + ✨: Gemini-refined label
• Dashed box: Predicted position (object not currently detected)
• Glow effect: Active detection

• 🚦 Traffic lights
• 🛑 Stop signs
• 🚗 Vehicles (cars, trucks, buses)
• 🚶 Pedestrians

• 🚦 Walk/Don't Walk signals
• 🛑 All traffic signs (stop, yield, speed limit, etc.)
• ⚠️ Road hazards
• 🚧 Construction zones
• More accurate labels

python server.py  # Backend on :8000
python -m http.server 8080  # Frontend on :8080

Option 1: Backend on Render + Frontend on Vercel

1. Deploy Backend (Render/Railway/Heroku):

# Push to GitHub
git push origin main

# On Render.com:
# - New Web Service
# - Connect repo
# - Build: pip install -r requirements.txt
# - Start: python server.py
# - Add environment variable: GEMINI_API_KEY

2. Deploy Frontend (Vercel/Netlify):

# Update script.js config.apiEndpoint to your backend URL
# Then deploy to Vercel
vercel

Option 2: Single Server

• Deploy entire app to one server
• Backend serves API + static files
• Simpler but less scalable

Edit script.js:

config: {
    apiEndpoint: 'https://your-backend.onrender.com/analyze',
    processingInterval: 100,  // COCO-SSD speed (10 FPS)
    geminiInterval: 2000,     // Gemini frequency (0.5 FPS)
    minConfidence: 0.3        // Detection threshold
}

Adjust for your needs:

• Faster COCO: Lower processingInterval (more CPU)
• More Gemini: Lower geminiInterval (more API calls)
• Less noise: Increase minConfidence

┌─────────────────────────────────────────┐
│  📱 Camera View                         │
│                                         │
│    ┏━━━━━━━━━━━━━━┓                    │
│    ┃ 🚦 Traffic    ┃ ← COCO-SSD        │
│    ┃    Signal     ┃                    │
│    ┗━━━━━━━━━━━━━━┛                    │
│                                         │
│    ┏━━━━━━━━━━━━━━━┓                   │
│    ┃ ✨ Walk Signal ┃ ← Gemini refined │
│    ┃    - Green     ┃   (thicker glow) │
│    ┗━━━━━━━━━━━━━━━┛                   │
│                                         │
│    ┏ ┄ ┄ ┄ ┄ ┄ ┄ ┓                    │
│    ┆ 🛑 Stop Sign  ┆ ← Predicted      │
│    ┗ ┄ ┄ ┄ ┄ ┄ ┄ ┛   (dashed)        │
└─────────────────────────────────────────┘

Gemini API (Free tier):

• 15 requests per minute
• 1,500 requests per day
• ~$0.01 per 100 requests after free tier

Usage:

• 0.5 requests/second = 30 requests/minute
• ~1,800 requests/hour
• Should stay within free tier for testing!

• Check backend is running (python server.py)
• Verify GEMINI_API_KEY in .env
• Check browser console for API errors
• Confirm config.apiEndpoint is correct

• Increase processingInterval (lower FPS)
• Increase geminiInterval (less refinement)
• Use better device/browser

• Enable device motion sensors in settings
• Keep device steady during initial detection
• Check AR tracking parameters in code

• Frontend: Vanilla JS (PWA)
• Fast Detection: TensorFlow.js + COCO-SSD
• Accurate Refinement: Google Gemini 2.0 Flash
• Backend: FastAPI (Python)
• AR Tracking: Custom (IoU, Kalman-like prediction)
• Camera: WebRTC getUserMedia
• Audio: Web Speech Synthesis
• Storage: IndexedDB

• COCO-SSD: 100% local, no data sent
• Gemini: Images sent to Google for refinement (optional)
• No Tracking: No analytics, no user data collection
• Works Offline: COCO-SSD continues without internet

# Install dependencies
pip install -r requirements.txt

# Run with auto-reload
uvicorn server:app --reload --port 8000

# Run frontend
python -m http.server 8080

Built with:

• TensorFlow.js
• COCO-SSD Model
• Google Gemini API
• FastAPI

Made for accessibility. Empowering visually impaired users with real-time hybrid AR detection.

🌟 Star this repo if you find it useful!