DeFakeIt

DeFakeIt is a Chrome extension that uses dual MesoNet CNNs to detect deepfakes from image screenshots or uploads and provides real-time confidence scores to combat AI-generated misinformation.

Comment

Inspiration

Deepfakes (AI-generated images that look real) are rapidly becoming more prevalent in our everyday media. Nearly 50% of voters said that deepfakes had an influence on them in the past election. We wanted to combat this rise of misinformation with a real-time solution to spot fakes.

What it does

DeFakeIt is a Google Chrome extension that empowers users to verify the authenticity of images that they find online. With just a screenshot, drag-and-drop, or file upload, the extension analyzes the image to provide a real-time verdict on whether it is an authentic image or AI-generated deepfake. The extension provides instant visual feedback with a confidence score, giving users the tools they need to navigate content online.

How we built it

Frontend: The Manifest V3 Google Chrome extension is the interactive frontend for this tool, built with JavaScript, HTML, and CSS. We chose to make this a browser extension to seamlessly integrate our tool with the browsing experience and implemented content scripts and the chrome.scripting API to have a seamless UI, between our screen clipping feature, drag-and-drop file uploads, clipboard-past, and toolbar badge directly in the browser. Backend: For the backend, we created a Flask server in Python that processes and analyzes images using MesoNet, a CNN architecture designed specifically for deepfake detection. MesoNet works by examining the subtle inconsistencies that often appear in deepfake content. There are two MesoNet variants that we worked on. First, we have MesoNet4 which is relatively simple. This model is optimized for quick performance. The second model was MesoNetInception4 which is more sophisticated and allows different features at various scales to be captured simultaneously. This helps identify deepfakes with less obvious discrepancies. The backend is hosted on Google Cloud Run.

We used a RESTful API to connect the front and backend. Images are transmitted and analyzed in real-time as the server returns a confidence score and classification that is immediately displayed to the user.

Challenges we ran into

A major challenge was the steep learning curve-we were working with many new tools and technologies, which we had to learn and apply on the fly. We encountered challenges with false positives, especially when detecting more modern deepfakes. One limitation we observed was that the MesoNet models were not trained on the latest generation of deepfakes, which are significantly more difficult to detect.To address this, we iteratively refined both our model by training it with newer and larger datasets. We also had a bug where Mac stores AirPlay at port 5000, so we had to run python server –5001 so we didn’t override it.

Accomplishments that we're proud of

Keeping the tool fast and responsive while implementing a complex model was a technical challenge. We are also proud of our intuitive UI that makes this complex detection tool accessible to everyday users. The extension’s ability to process and analyze an image in under 2 seconds demonstrates our feat of performance and usability.

What we learned

Hands-on experience with training and fine-tuning MesoNet Convolutional Neural Network machine learning model architectures for deepfake detection How to use Flask, Docker, Google Cloud, and build a Google Chrome extension Containerized our Flask REST API with Docker, deploying it to Google Cloud Run How to bridge a complex backend with a usable frontend How to build in features like screen clipping, drag-and-drop file upload, and copy & paste image ingestion. Researched and evaluated available SDKs and deepfake detection model options for evaluation

What's next for DeFakeIt

Up next, we’re planning on publishing our Google Chrome Extension so users can one-click download to add DeFakeIt to their browser. We also want to gamify the extension, so users can submit flagged images to add to our database, and get one point for each valid find which will count into our (optional) weekly leaderboards. We’ll use this data to continue to retrain our MesoNet Keras Convolutional Neural Network model to improve its performance and extend our support to real-time video streams as well. Another idea we had was to develop a public API, which would allow other third-party platforms and services to leverage our technology.

Built With

Share this project:

Updates