Multimodal Misinformation Detection

Inspiration

We were inspired by the current upheaval in leadership in Twitter, due to the rampant problems with misinformation and bias on the site.

What it does

We utilized state-of-the-art Machine Learning methods to take in 2 pieces of text corresponding to similar images and finding inconsistencies between the two, in order to detect misinformation.

How we built it

We utilized a dataset from Twitter's photo storage system called the Photo Blob Storage (PBS for short). We then used text preprocessing methods from the BERT model, as well as some custom ones. Using Resnet as our pre-trained vision model, and BERT as our pre-trained text processing model, we created a multimodal misinformation detection model achieving testing accuracy of 85%+.

Challenges we ran into

We ran into challenges downloading the dataset as the wi-fi was very slow.

Accomplishments that we're proud of

We are proud of our high accuracy with such a complicated, nuanced task.

What we learned

We learned about state-of-the-art methods in multimodal machine learning.

What's next for Multimodal Misinformation Detection

We plan to use more advanced web-scraping methods for our input for the user, as well as training on larger, more diverse set of samples to allow for a more impactful model.

Built With

• python