Scam Scare

Elders in our society lose $11B+ to financial fraud every year, virtually all over phone. We built a spooky AI assistant to vet unknown numbers and get scammers off the streets for good.

Comment

Inspiration

On the drive to Denton, we were talking about our shared experience of growing up having taken care of our grandparents living in close proximity and seeing repeated attempts of financial fraud on them over the years. As soon as we learned about HackUNT's core theme, we realized that spooking scammers for good would also be the perfect hackathon idea (with real-world applications!) to work on.

What it does

Scam Scare includes an AI assistant that intercepts unknown callers to inquire about their reason for calling. If anything remotely financial, it:

  1. verifies them working at the institution they suggest they do
  2. walks them through a security questionairre that verifies they're calling from a place Jane currently has an account at
  3. scans their carrier details for association with any previous scandals

If they're unable to clear this check, they're SPOOKED with a scream effect and their information is displayed on a public dashboard that carriers and federal fraud agencies can use to act upon immediately.

If the unknown number does clear the check, Scam Scare records the call and monitors to make sure nothing fishy is going on.

With carrier help, it'll also ideally auto-suspend them so they can no longer harm others with the same number.

How we built it

  • We built the core voice sequence using SignalWire and their custom SWML language / endpoint requests integrations.
  • Routed voice prompts to Nicole at ElevenLabs (the absolute creepiest voice we could find!).
  • Use Perplexity's API for the script that checks caller's current LinkedIn status
  • Built a Custom LLM Flow to verify using Jane's given details (last 4-digits, bank names, etc.)
  • Used NextJS and some Vanilla JS for the frontend
  • Firebase Storage for the Backend Database that verifies scam calling status
  • Firebase Cloud Functions to deploy the functions that change database values
  • Twillio's APIs to find Caller Carrier, Location, and SMS Scam Risk Score

Challenges we ran into

When initially trying to intercept calls with Twillio (and transcribing them to connect to ElevenLabs / OpenAI's new realtime API), we made significant headway into building the project until we realized that the feature we needed to connect it all together (i.e. Twillio's Flex MediaStreams) were only available for enterprise customers on a week+ notice. Fortunately we were able to pivot quickly into building with SignalWire instead.

Accomplishments that we're proud of

As aspiring software engineers relatively new to hackathons, we were incredibly proud of being able to build what we had initially envisioned, regardless of how much we had to pivot technologies and how late we had to stay up. We're also really, really excited about the potential real-world use case for our tech!

What we learned

We learnt how to build fully integrated apps, work with caller APIs, how to test conversation flows with LLMs to make them more robust, using voice APIs from vendors such as ElevenLabs, and more about all the useful information we can obtain from public APIs.

What's next for Scam Scare

We're genuinely passionate about this problem space (elders in our immediately family lost thousands of dollars to financial fraud just last year) and would love to see if there's a way to productize this as a nonprofit offering. Our plan is to turn the voice scripts into a lightweight mobile app that concerned adults can download for their elderly parents to intercept unknown calls.

The step right after would be to work with the largest telecom providers (like AT&T & T-Mobile) to auto-suspend high-risk callers and bring our dashboard to the FTC's Fraud Division.

Built With

Share this project:

Updates