ScamShield

AI that listens, thinks, and warns —ScamShield detects scam calls in real time, transcribes speech, analyzes intent, and alerts users instantly, protecting them from fraud before they’re tricked.

Comment

Inspiration

Every day, millions of people receive fraudulent calls pretending to be their bank, the IRS, or tech support. These scams manipulate human trust and cost billions of dollars globally each year. While email spam filters and browser warnings have matured, voice communication remains unprotected, there’s no “scam filter” for phone calls. We wanted to fix that. Our goal was simple: restore digital trust in voice communication by combining AI, telephony, and cloud technology into a single, real-time defense layer that listens, understands, and warns before damage occurs. That’s how ScamShield AI was born, an AI guardian that protects users during live phone conversations.

What it does

ScamShield AI detects voice-based scams as they happen. When a call begins, Twilio Media Streams sends the live audio to our FastAPI backend. Audio is processed, cleaned, and transcribed in real time using OpenAI Whisper. The transcript is passed to a machine learning pipeline that analyzes linguistic intent, tone, and keyword patterns to detect scam-like behavior. If the model detects suspicious language, such as requests for banking info, OTPs, or emotional manipulation, ScamShield instantly sends a real-time risk alert to the user interface via WebSocket.

The system displays: ⚠️ Scam Type (e.g., “IRS Impersonation” or “Tech Support Fraud”) 📊 Risk Score 🧠 AI Explanation of why it was flagged All audio, transcripts, and ML feedback data are securely stored in Cloudflare R2, enabling transparent retraining and continuous improvement.

How we built

Core Backend

FastAPI + Uvicorn: Asynchronous framework powering WebSockets and REST APIs

Twilio Media Streams: Live voice data from phone calls

OpenAI Whisper: Real-time speech-to-text transcription

scikit-learn + Sentence-Transformers: Hybrid classifier using contextual embeddings

RapidFuzz: Fuzzy matching for scam keyword patterns

SQLite: Fast local data logging and caching

Cloudflare Ecosystem (Key Integrations)

Cloudflare R2 – Hybrid Object + Vector Storage Stores audio recordings, transcripts, and a vector database of call embeddings. Used for feedback loops: retrains and improves the ML model based on misclassifications and new scam types. Enables semantic similarity search for detecting recurring scam patterns across users.

Cloudflare Workers AI (Prototype Extension) Edge inference to process scam detection faster, reducing latency for real-time alerts. Moves partial ML tasks (keyword extraction + confidence calculation) to Cloudflare’s edge network.

Cloudflare Tunnel Provides secure, zero-config HTTPS exposure for Twilio webhooks and WebSocket communication during development. Eliminates need for public IPs or local port forwarding — instant, encrypted connection between Twilio → FastAPI → R2.

Cloudflare Analytics Edge Service (Custom) Aggregates usage metrics, fraud attempts, and model confidence trends via Workers KV. Displays on our internal dashboard to visualize fraud attempt rates, model accuracy, and latency heatmaps across geographies.

This full Cloudflare integration turned ScamShield from a basic ML tool into a globally distributed, low-latency AI system.

Frontend

The frontend was designed for simplicity, accessibility, and responsiveness — ensuring ScamShield feels like a consumer app anyone could use. Built using a React + Tailwind + MUI stack for fast and beautiful UI, and powered by real-time WebSocket data, the interface visualizes scam alerts the moment they’re detected.

Tech Used

React + React-DOM – Component-based frontend architecture. Tailwind CSS – Utility-first CSS for rapid styling. MUI (Material UI) – Modern design system for clean, consistent components. Aceternity UI – Used for modern alert components and animations. HTML5 + JavaScript (ES6) – Core frontend logic and dynamic updates. Zod – Schema validation for frontend inputs and API responses. WebSocket Client – Live data stream from backend alerts.

UX Highlights

Real-time dashboard showing risk level, scam category, and transcript snippet. Smooth animations via Acerenity UI for alert notifications. Responsive layout built with Tailwind and MUI Grid for mobile + desktop. Zod-powered form validation for safe API testing and call simulation.

Challenges we ran into

Managing 5-second streaming audio buffers without desynchronizing transcription. Keeping latency under 1 second while handling audio → text → AI → alert pipeline. Integrating Twilio’s real-time media with Whisper’s synchronous API. Designing a context-aware ML model that doesn’t rely only on scam keywords but understands intent and tone. Building secure, global storage using R2 + Tunnel + encrypted endpoints. Implementing vector search feedback loops without bloating storage or breaking R2 indexing.

Accomplishments that we're proud of

Created a fully functional real-time scam detector that works during live calls. Integrated Cloudflare’s full stack (R2, Tunnel, Workers, KV Analytics) for secure, global scalability. Designed an ML feedback loop with vector storage to improve model accuracy over time. Developed a real-time WebSocket dashboard with risk alerts and live transcript display. Built a system that can scale from personal protection app → enterprise SaaS API in one architecture.

What we learned

How to orchestrate streaming AI pipelines under tight time constraints. The power of Cloudflare’s developer ecosystem for serverless deployment, storage, and analytics. That consumer trust can be rebuilt through intelligent, transparent AI tools. The challenges of maintaining contextual accuracy in scam classification — and how vector feedback systems can fix that. How to design software that balances AI performance, cloud scalability, and human usability in one flow.

What's next for ScamShield

Launch ScamShield API for telecoms and fintechs to detect fraudulent calls enterprise-wide. Build a mobile app + Chrome extension for consumers to receive live scam alerts on their devices. Train the ML model on multi-lingual and emotional manipulation datasets for higher global coverage. Deploy Workers AI inference for sub-200ms latency fraud detection at the edge. Open-source a public Scam Transparency Portal where anonymised scam call data is shared to raise awareness. Our long-term goal is to make ScamShield the “spam filter” for voice communication — a universal trust layer for the world’s phone networks.

Built With

Share this project:

Updates