Professor Garmin

Video player on upload
Textbook excerpts

Inspiration

Having skipped a multitude of classes but being fortunate enough to have professors that record and upload their lectures, we got annoyed when we had to constantly pause recordings to pull out textbooks and lectures notes to full understand the concepts the professors were talking about. This is why we wanted to centralize all the activity on one platform, hence Professor Garmin.

What it does

Professor Garmin takes a course textbook as well as any lecture recording (so long as the topic of these lectures corresponds with the course textbook), and returns a clipping station where students can watch video and clip at any moment if there's a concept that is difficult to understand. Garmin then automatically brings out any related excerpts from the textbook and pastes it to the side of the video for students to reference.

How we built it

For the semantic search, we first broke down textbooks using HuggingFace's SentenceTransformer text embedding generation model. These vectors are then stored in Pinecone's vector DB for search. While watching the video, if a student decides to "clip" a moment, Garmin takes a clip of about 30 seconds and transcribes the audio through OpenAI's whisper model. The transcription is then used to query the pinecone db for any

Challenges we ran into

There were a lot of components to get the product working and to implement all these components, we ended up producing a lot of poor quality code which came to bite us back when it came to debugging time. We spent a lot of time debugging.

Accomplishments that we're proud of

We're proud to have implemented the semantic search to pull textbook content from video content.

What we learned

We learned the general process of setting up the data required to perform a semantic search, as well as the various free technologies to achieve the task. We also learned how to route the application via react-router.

What's next for Professor Garmin

A next step for Professor Garmin is to have multi-language support. We hope to be able to translate video dub into an audio of any language students may need. This would extend to the textbook excerpts as well.

Additionally we hope to be able to implement a feature that takes any professor and their teaching style to teach any content at all.