fishy.watch: 2FA (mobile) network against deepfake phishing

What it does

fishy.watch is a mobile app designed to protect loved ones against deepfake phishing by establishing a P2P network of trusted users and their devices (phones). Users establish connections by using an NFC/Proximity key exchange and storing mutual voice signatures. Then, the fishy client detects signs of phishing attacks based on voice signatures and network context state, and warns the user if there is a risk of phishing. It features a privacy-first local inference design, with options for cloud defence mode for enhanced phishing security.

Inspiration

Deepfakes are no longer distinguishable from reality. Deep research tools enable stalking at an unprecedented scale, allowing for highly tailored and targeted impersonation and spear phishing attacks. Now, we humans are the weakest security link.

Imagining my mum and loved ones being emotionally manipulated and phished by a deepfake of me terrifies me. However, they aren't as tech-savvy as I am. We NEED a solution for this NOW.

That's why we built fishy.watch, 2FA network to fight against phishing, first by tackling a niche of deepfake phishing calls, but starting to build contextual information about the network of loved ones.

How We Built it / Technical Design

The app is built primarily on Expo React Native, built on Bolt.new.

2 protection modes available for different types of users: 1) Local (Privacy) Mode: P2P design with all data E2E encrypted, all inference (voice signature match, which is cosine similarity) happens locally. (mostly react native features) 2) Cloud Defence Mode: More intelligent security solution on the cloud, powered by an ElevenLabs audio agent for real time context-aware phishing protection (ElevenLabs + Supabase Edge Functions + Entri)

DeepFakes were built using ElevenLabs voice closer of my own boice.

See diagrams attached & the demo video to see how each mode works in detail.

NFC/Proximity based key exchange carried out to enable P2P network communication, with full privacy preservation, so all the data is kept between users.

Cloud defence mode was built with Supabase + Elevenlabs. It is obviously more intrusive, but it provides more nuanced and robust security.

Challenges

• Writing platform-native code, since Expo/react native doesn't support many of the native Android/iOS features, like finding nearby devices, checking call state, and NFC key exchange.
• Walled garden: neither Android nor iOS allow easy access to listen in to calls. Thus, we need to use a Bluetooth device to intercept the audio (raspberry pi)

What we learned

• Challenges with implementing mobile-native features (e.g. NFC, proximity checks, call state checks)

What's next for fishy.watch

The fishy.watch solves the deepfake phishing problem in deepfake calls, but phishing is more multi-channel, and we need a more comprehensive solution to solve all phishing. We believe the fishy.watch concept can be extended to everyone in the world who is connected to the internet, and many businesses will want a full-package solution against phishing.

Keywords: Voice signature RAG, 2FA network, P2P network, fingerprint auth

Built With

• android
• bolt.new
• elevenlabs
• supabase