Health monitoring for elderly care using Meta Ray-Ban smart glasses. The glasses periodically record audio and video, which is analyzed by multiple AI services to assess the wearer's wellbeing and screen for cognitive decline.

Meta Ray-Ban Glasses
        │
   DAT SDK v0.4.0
        │
┌───────▼────────┐     ┌──────────────────┐     ┌──────────────────┐
│  Android App   │────▶│  FastAPI Backend  │────▶│  Next.js Dashboard│
│  (Kotlin)      │     │  (Python)        │     │  (TypeScript)     │
└────────────────┘     └───────┬──────────┘     └──────────────────┘
                               │
                 ┌─────────────┼─────────────┐
                 │             │             │
           ┌─────▼───┐  ┌─────▼───┐  ┌──────▼─────┐
           │Modulate  │  │  Reka   │  │  Pioneer   │
           │(Sentiment│  │ (Video) │  │(Alzheimer's│
           │ Analysis)│  │         │  │ Screening) │
           └─────┬────┘  └────┬────┘  └──────┬─────┘
                 │             │              │
                 └──────┬──────┘              │
                   ┌────▼────┐                │
                   │ OpenAI  │                │
                   │(Overall │                │
                   │ Health) │                │
                   └─────────┘

1. Phase 1 (parallel):
   • Modulate AI — Sentiment analysis on audio (emotions, valence, arousal)
   • Reka AI — Video frame description (activity, environment, posture, concerns)
2. Phase 2 (parallel):
   • OpenAI GPT-4o — Combines sentiment + video into overall health score (0-100)
   • Pioneer AI — Alzheimer's screening from audio + behavioral context

All services have mock fallbacks so the system works without API keys during development.

Kotlin app using Meta DAT SDK v0.4.0 to connect to Ray-Ban glasses.

• GlassesManager — SDK initialization, device discovery, camera streaming
• AudioRecorder — PCM→WAV recording at 16kHz mono
• ScheduledRecorder — 5-min interval recording (30s clips), parallel audio+video capture
• MonitoringService — Foreground service for background operation
• MetricsLogger — Performance metrics (camera turn-on time, recording delay, upload duration)
• ApiClient — Multipart upload to backend

1. Copy local.properties.example → local.properties
2. Add your GitHub PAT (for Meta DAT SDK access) and server URL
3. Build with Android Studio (minSdk 29, compileSdk 34)

Python FastAPI server that orchestrates the AI pipeline.

cd backend
cp .env.example .env    # Add your API keys
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

Or use the run script:

chmod +x run.sh && ./run.sh

Next.js 14 dashboard with dark theme showing real-time health data.

• Dashboard — Patient grid with health scores, auto-refresh every 15s
• Patient Detail — Sentiment analysis, video observations, health assessment, Alzheimer's screening
• Charts — Health score trends via Recharts

cd frontend
cp .env.local.example .env.local
npm install
npm run dev    # Runs on port 3000

The frontend proxies API requests to localhost:8000 via Next.js rewrites.

Without API keys, all AI services return mock data so you can develop the full pipeline locally:

# Terminal 1 - Backend
cd backend && source venv/bin/activate && uvicorn app.main:app --reload

# Terminal 2 - Frontend  
cd frontend && npm run dev

Open http://localhost:3000 to see the dashboard.

The Android app logs detailed timing metrics:

• sdk_initialization — DAT SDK startup time
• camera_turn_on — Time to activate glasses camera
• stream_initialization — Stream session setup time
• first_frame_received — Latency to first video frame
• photo_capture — Photo capture duration
• audio_recording — Audio recording duration
• upload_total — Server upload time
• recording_cycle_total — Full cycle (record + upload) time

Access via the Metrics button in the Android app.