Kova

The elderly have sacrificed so much for our sake, yet they are the most vulnerable to soulless attackers who prey on their ignorance. It’s time for a companion to protect them against scammers: Kova.

Comment

Inspiration

Scams targeting older adults are rapidly escalating. According to the US Federal Trade Commission, in the United States, adults aged 60 and over reported losing approximately $2.4 billion to fraud in 2024, more than four times the losses reported in 2020. Many cases are never reported, meaning the true financial impact is likely far higher. At the same time, phone-based scams are becoming more convincing as attackers increasingly use AI-generated voices and impersonation tactics to pose as banks, government agencies, tech support, and even family members.

Despite how common and dangerous scam calls have become, there are almost no tools that are a companion to the elderly during the potential scams. Most existing protections focus on blocking or labeling calls before they are answered, such as services like Nomorobo or RoboKiller, which rely on known spam numbers and call pattern detection. While these tools can reduce the volume of unwanted calls, they do not analyze the content of a live conversation or account for situations where scammers get through, leaving users unsupported during moments of pressure, urgency, and manipulation.

We built Kova to fill that gap. Kova sits alongside the user during live calls, analyzing the conversation in real time, suggesting verification questions, providing guidance through an AI chat companion, and alerting close family members when something seems wrong. Instead of relying on hindsight or education alone, Kova provides protection in the moment it matters most.

What it does

Kova is a real-time scam call protection system that:

  • Listens passively using a voice-activated wake word ("kova, activate") so users can start protection hands-free
  • Two-stage transcription pipeline: Deepgram handles ultra-low-latency streaming transcription for real-time reaction, while Llama performs fast semantic analysis on the transcript to distinguish between user and caller roles
  • Analyzes conversations with an LLM-powered "Scam Brain" that maintains context across the entire call and outputs a dynamic risk score and confidence
  • Generates verification questions to help users interrogate suspicious callers (e.g., "Ask for their employee ID and call-back number")
  • Sends instant iMessage alerts to close family when a scam is detected with high confidence
  • Provides an AI chat companion (Claude 3.5 Sonnet) that is directly connected to the Scam Brain, allowing users to ask questions mid-call and receive guidance grounded in the live call context
  • Reports suspicious numbers to a community database for pattern detection and warns users if an incoming or active call involves a known malicious number
  • Shows a live analytics dashboard with call history, risk trends, and visualizations

How we built it

  • Keywords AI Integration ensured we could iterate upon system prompts efficiently and choose the best models for different tasks
  • Frontend: React + Vite + TypeScript with Tailwind CSS and Framer Motion for smooth animations
  • Backend: FastAPI (Python) handling WebSocket streams for real-time audio
  • Transcription: Deepgram SDK with Nova-2 model for sub-200ms latency
  • AI Pipeline: LangGraph orchestrates our multi-step workflow, scam detection (Groq/Llama 3.3 70B), question generation, and alerting logic
  • Chatbot: Claude 3.5 Sonnet (via Keywords AI) for intelligent mid-call assistance
  • Database: Supabase (PostgreSQL) for user profiles, emergency contacts, and suspicious number tracking
  • Alerts: Native macOS AppleScript to send iMessages directly

Challenges we ran into

  • Wake word detection was unreliable with Web Speech API, so we rebuilt it using Deepgram with phonetic fuzzy matching for "kova" since it's not a real word
  • Real-time latency required aggressive tuning of audio chunk sizes (250ms) and WebSocket buffering
  • Session state management across multiple concurrent WebSocket connections was tricky until we implemented a proper session manager
  • Twilio verification takes days, so we pivoted to macOS native messaging as a workaround for the demo

Accomplishments that we're proud of

  • Achieved sub-second end-to-end latency from speech to on-screen risk score update
  • Built a beautiful, polished UI with real-time animated visualizers that feels production-ready
  • The LangGraph workflow elegantly handles conditional routing, alerting only when both risk AND confidence are high
  • Successfully integrated 3 different AI models (Deepgram, Llama 3.3, Claude 3.5 Sonnet) into a cohesive experience

What we learned

  • WebSocket architecture patterns for real-time bidirectional streaming
  • LangGraph as an orchestration framework for multi-step AI workflows
  • The importance of graceful degradation (e.g., switching from Twilio to native messaging)
  • Phonetic matching strategies for detecting made-up words in speech recognition

What's next for Kova

  • Mobile app with call integration for Android/iOS
  • Multi-language support for protecting non-English speakers
  • Partner with senior care organizations to get Kova into the hands of those who need it most
  • Dedicated physical device that sits in the home, listens during calls, and provides ambient visual or voice feedback based on confidence level

Built With

Share this project:

Updates