YouTube Mentions

Uses named entity recognition. Then queries wikipedia for info on the subject
Uses shazam for song recognition and info. Then queries wikipedia to find info about the artist
Uses pyppeteer for scraping YT page to learn game played. Then queries wikipedia for info about game

Inspiration

Consuming media can be extremely difficult without the right context. Information can be misinterpreted creating a barrier between the content creator and the user.

What it does

Our application analyzes the given YouTube video and retrieves quick snippets of relevant information about people, songs, games, etc. mentioned in the video and displays them side-by-side.

How we built it

The frontend was built using Svelte while the logic of retrieving relevant information is in python. We run spaCy's named entity recognition on the YouTube transcript and make resulting queries to Wikipedia. We also used Shazam for song recognition.

Challenges we ran into

Repeatedly sending requests through python wikipedia package greatly increased loading time. We replaced this with our own requests and strategically discarded less important data to cut down loading time significantly.
Initially, we had no idea how to scrape a JS rendered page; Eventually, we found out that pyppeteer (an unofficial python package for puppeteer) could wait for all JS sources to load before scraping the page.
Many YouTube videos didn't have ideal transcripts; Often, they were in all uppercase, all lowercase, and some videos had trancripts disabled, resulting in lower accuracy for entity recognition. To be improved in the future.

What we learned

We learned to scrape from a page rendered with JS. We learned how Named Entity recognition works and how its limitations apply to our project. We learned to use Svelte to dynamically render content.

What's next for YouTube Mentions

Ability to customize settings that affect which information and level of detail users see.
Stream the relevant data as user is watching instead of retrieving all information before user can watch.
Improve on accuracy of named entity recognition
Add support for languages other than English