PhishReport

Our data pipeline

Inspiration

Phishing emails continue to be one of the most pervasive and dangerous cyberattacks used by hackers. We created PhishReport to empower organizations in the fight against phishing emails by providing insights into trends and commonalities in these emails. Phishing isn't going anywhere, but the insights offered by PhishReport can be leveraged by organizations to create more robust plans for dealing with this ever-present threat.

What it does

PhishReport collects and analyzes confirmed phishing emails, and presents the collected data in a dynamic dashboard, revealing underlying trends and patterns. PhishReport's processing and analysis includes extracting data from the email headers, subject, and body, and running sentiment analyses on the emails' content using a machine learning model.

How we built it

PhishReport is made up of four distinct components. First, a python script runs on a scheduled basis, fetching new emails before extracting the data we want to capture. Next, this data is stored in a PostgreSQL database. The frontend is a static React website. An API built in python with FastAPI sits between the database and the frontend, listening for requests and serving records. The app is hosted by Render.com, which uses docker to containerize the backend.

Challenges we ran into

The biggest challenge we faced creating PhishReport was integrating all of the separate parts to create a continuous data pipeline from email ingestion to data visualization on the frontend. None of us had ever put a full stack app together before, so making sure that each layer could talk to its neighbors was a huge effort. Another major challenge we faced was dealing with the messy data that emails can contain. Like most real-world data, there was a huge number of edge cases we had to account for and find ways to deal with.

Accomplishments that we're proud of

We are very proud of how much we were able to collaborate and learn together. We set goals for ourselves and held informal sprints to share and track our progress. We did a great job in supporting each other, both in technical and moral support. We are also proud of our technical achievements this weekend. Seeing each of our distinct projects transform into one unified application was incredibly satisfying, especially because integration was our biggest concern throughout the hackathon.

What we learned

Each of us learned so much over the course of this hackathon, across a variety of domains. We learned about hosting and integrating a live application, connecting to APIs (both external APIs like gmail's and our own API), and using machine learning models to process natural language. We also learned a ton about the process of putting together an app like this. We did a good job of splitting up responsibilities, allowing different parts of the app to be built in parallel, and this took a huge amount of teamwork and communication.

What's next for PhishReport

We would love to add more information and interactivity to the frontend dashboard. One specific element we would like to add to PhishReport is the ability to plot data over time, seeing how phishing emails and their authors change and adapt as the cyber-landscape continues to shift and evolve.