AI-powered real-time vision assistance for the visually impaired, built at DartHacks 2026.
Sixth Sense gives visually impaired users a real-time AI narrator for their surroundings. Point your phone camera and Kayla — an AI avatar — describes what she sees and guides you through your environment using natural voice.
Core features:
- Live vision narration — Camera captures your surroundings every 15 seconds, Claude AI describes what it sees
- Navigation guidance — Kayla tells you what's ahead and which way to turn
- Voice commands — Hands-free control with "Hey Kayla" wake word
- SOS — One tap calls 911 with a 15-second countdown
- Emergency contacts — Instantly SMS your contacts when you need help
- AI Avatar — Lip-synced avatar powered by HeyGen LiveAvatar
| Command | Action |
|---|---|
| "Hey Kayla update" | Immediate camera analysis |
| "Hey Kayla pause" | Pause narration |
| "Hey Kayla resume" | Resume narration |
| "Hey Kayla SOS" | Trigger SOS screen |
| "Hey Kayla help" | Alert emergency contacts |
| "Hey Kayla cancel" | Cancel SOS/help countdown |
| Layer | Technology |
|---|---|
| Frontend | React + Vite, deployed on Vercel |
| Backend | Node.js + Express, deployed on Railway |
| AI Vision | Claude claude-sonnet-4-6 (Anthropic) |
| Avatar | HeyGen LiveAvatar + LiveKit WebRTC |
| Database | MongoDB Atlas |
| SMS Alerts | Twilio |
- Node.js 18+
- MongoDB Atlas account
- API keys: Anthropic, HeyGen, Twilio
git clone https://github.com/Aryav-T/Sixth-Sense
cd Sixth-SenseFrontend:
npm install
npm run devBackend:
cd server
npm install
cp .env.example .env # fill in your API keys
node server.jsCLAUDE_API_KEY= ELEVENLABS_API_KEY= HEYGEN_API_KEY= MONGODB_URI= TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN= TWILIO_PHONE_NUMBER= PORT=3001
Best experienced on mobile Safari or Chrome. Grant camera and microphone permissions when prompted.
Built by Aryav and Vrishank at DartHacks 2026.