Scuttlebutt

extracting valuable financial trend insights from social media

Comment

Inspiration

There's so much valuable data on the internet. It's impossible to pay attention to all of the trends.

What it does

Scuttlebutt is a system to extract financial trends from social media and news sources. What are people talking about

How we built it

We used Reddit's API to get the raw data we needed, Alpha Vantage API to get stock market data, and CryptoCompare API to get cryptocurrency market data. We stored the raw data in MongoDB clusters. The Reddit data is processed using a Python Natural Language Processing (NLP) package called spaCy which we used to get information about which cryptocurrencies and stocks people are talking about. The webapp was developed using NodeJs / Express.js with data visualization using d3.js.

We built Scuttlebutt with scalability in mind. With the modularized framework we built up, we have the capability to easily integrate more media sources alongside Reddit, store it in our MongoDB database, and present it on our webapp.

The applications for the Scuttlebutt system are vast. One particular idea we're excited about is integrating our extracted market trend data into an application which notifies users about emerging market trends so that they can take advantage of and capitalize with a "first-mover advantage".

Challenges we ran into

Time constraints, small team size (just the two of us!), amount of data we had to collect to get valuable knowledge, processing time / processing power limitations, learning and incorporating multiple different APIs, learning Express.js web application framework in <24 hrs.

Accomplishments that we're proud of

Building a clean MongoDB database that is connected to multiple APIs and a web application. We're proud of our concept we developed, its level of complexity, and how we future-proofed it / made it scalable. We worked really hard with very little breaks.

What we learned

Finance data is hard. Date management is difficult. Learned a lot about Node.js web applications, learned new python packages (spaCy), and got better at working with APIs.

What's next for Scuttlebutt

Keep polishing the data visualization, adding functionality to the site, hosting the site on a server, doing more data processing/knowledge extraction, and adding more media sources to pull data from!

Share this project:

Updates