Zazu: AI Teaching Assistant

An AI generated quiz question using Zazu
Students can answer AI generated quiz questions and see the correct answer in real time

Inspiration

The main inspiration of the idea was in our own personal experience when we are in the middle of a live lecture or watching a pre recorded zoom lecture, and we had a question regarding the topic. The process of googling a result one the side while the lecture is going on is usually a herculean task that involves managing multiple tabs and usually always falling behind on the notes. Our website solves this problem.

What it does

Zazu is a video call platform designed for teachers and students. It allows students to have access to an AI TA right in their video call so that they can ask questions to the TA while not disturbing the teacher and the flow of the lecture. The teacher can also use a slew of AI powered features, such as generating quiz questions on demand to ask the students to check engagement. The teacher can also generate polls in lecture simply by asking Zazu to "generate poll" and a poll will be sent to all students instantly. At the end of the lecture once a student leaves call, a summary is created of the lecture with bullet point notes for the student, basically eliminating the need for students to manually take notes. The video call platform also supports all the features of Zoom and has AI features built on top of it.

How we built it

The front end of the website was built using the Dolby.io framework and React which allowed us to quickly implement most of the video call features. The backend is powered by Flask where we make multiple open ai requests to their Davinci model to get our results. We use Firebase Cloud Firestore to store our data and socket.io to communicate data between our server and client. We hosted our server on [server]. Most of the development revolved around building out features in React for the front end and building out API endpoints in Python on the backend. We heavily use OpenAI's Davinci LLM and a lot of time was spent prompt engineering to get the data that we require for each of our different features.

Challenges we ran into

The main challenge we ran into was getting the data from the Davinci LLM in a format that was consistent with every API call. We had to spend a large part of our time designing specific prompts for the LLM so that we get the response in the same format every time regardless of the topic of the lecture. We also had to spend a decent chunk of time on parsing the transcript of the lecture and filtering out meaningless remarks and noise in the data. To solve this, we had to use NLP filters in the backend and models to periodically summarize the transcript which makes it easier to send to the LLM to get results.

Accomplishments that we're proud of

We are extremely proud of the verbal speech-to-text commands that we built to help support teachers during their lecture. By simply invoking the command, "generate quiz", a multiple choice question about the topic of the lecture is generated by the Davinci LLM and instantly sent to all the students on call. This is powered by a slew of APIs and frameworks which is extremely impressive since our team is made up of mostly beginner hackers and this is most of our's first hackathon.

What we learned

The main concept we learned was how to build an end-to-end application with a complete feature set. We also got a significant amount of experience in using OpenAI's APIs and Dolby.io framework to create a front end video call app. Large portion of the time was dedicated to bug fixing and we learned a lot about version control and how to build a large project in parallel with different members contributing to different features.

What's next for Zazu

Zazu has the potential to become a great tool for educators around the world. The immediate next steps are to expand the feature set and make it complete with all the features of Zoom. We would also want to add support for using our feature set not only on video calls but also on pre-recorded lectures. Then we would ideally publish the app and market it to teachers, specifically college professors where this would largely come into play.