ScamBuster

Introducing ScamBuster, the ultimate defense against text message scams. With a simple scan, it reveals hidden scams. Stay safer with ScamBuster.

Comment

Hello! Welcome to our project, Scam Buster! With a swift scan, ScamBuster reveals hidden scams.

Reasoning behind idea

Our project was inspired by how many people we have seen be tricked by scam messages. In fact, my mother had actually come up to me the other day, showing me her phone, telling me about a delivery message she received that she wasn’t able to understand properly as she still struggled with English as a second language.

Turns out, it was nothing but a classic delivery scam message.

It is becoming more difficult to figure out what is and isn’t a scam message. As a result, there is a need for something that can accurately identify scam messages.

Introduction to Project & Goals

Utilizing the open-source LLM BERT built with PyTorch framework and Google colaboratory, we have developed an android app that allows users to check fraudulent messages.

While SMS message scams may not seem like much of a threat: In 2020 alone, text scams costed Americans 86 million dollars Only 65% of Americans said they would delete texts they received from an unknown number 21% of fraud reports that were filed in 2021 were through SMS.

And this isn’t only concerning the elderly population: People under the age of 39 fall for scams most often out of any age group, however those in their 60s end up lose the most money. Younger pupils often lost money 41% of the time their experienced fraud

Introduction to application

Clearly, SMS scams are a rampant issue. With this application, users are able to simply open the app and click on which message thread they want to have checked for scam activity.

There is no hassle of clicking a link, hoping it’s nothing bad and no need to ask others, “Hey, does this look like a scam to you?”.

There is no need to rely on a guessing game for your security.

This application allows users to safely check for scams without jeopardizing their safety, and with a 99.6% accuracy rate, it is not only useful but reliable.

Demonstration and functionality of application

Here is an overview of the application: Users open the app and are greeted with a list of numbers (usually sorted by most recent message sent) from their SMS app Users can search up a certain number By simply clicking the number, users can validate the messages to see if they are a scam or not

Information on Language Model

Here’s how we set up the language model: Our language model was created using Google colaboratory and the PyTorch framework. The dataset had a very uneven ratio of scam and ham messages, so I had to augment the data. Initially there were 700 scam messages, by the time I was done, there were 4825. We loaded a pretrained Mobile BERT classification model and the PyTorch framework to fine-tune the model. The BERT Tokenizer was used to tokenize the messages. Several training models were utilized, ranging from small (95mb) to large (400mb). Between the two ends, there was only a 0.2% difference in accuracy, so we went with the smaller model due to its convenience for mobile app creation. The accuracy ended up at a nifty 99.6%!

Trials, Errors and Learnings

Challenges: While working on this project, we came across a few issues.

To begin, none of us had had any prior experience with LLMs or Android app development, so we were in completely new territory. Finding the dataset was a big one, because the only available dataset was one with 5500 messages, which was not enough to bother fine-tuning an LLM, in case it overfitted. But the team managed to augment the data to increase to 10000 messages.

Unfortunately, we were unable to integrate the model within our app. First issue was, tokenizer was not currently being set up within our IDE. We even tried making our own tokenizer, which also failed. We set up a free trial account on GCP, but we were unable to deploy it due to some error (we were unable to pinpoint the issue). There was a tutorial using tflite models, but try as we might, converting our model gave many errors. We tried training the tflite model, but the free version of google colab kept crashing due to RAM Overload when the model was being saved.

Thank you for taking a look at our project!

Closing Remarks

We hope you found our project interesting! We plan on continuing our work with ScamBuster, and hope to be able to improve upon it. Thank you, once again, for watching.

Built With

Share this project:

Updates