CelebLearn

Inspiration

In a world where technology's potential is often misused, we couldn't help but feel inspired to change the narrative. While scrolling through TikTok, we were struck by the realization that powerful innovations like deepfakes were being used in less-than-ideal ways. It was then that we decided to embark on a mission to harness the potential of deepfakes for educational purposes.

What it does

Our innovative platform, CelebLearn, enables users to upload a PDF textbook image via their device, subsequently offering them the opportunity to select from a diverse range of celebrities. This unique feature allows users to receive personalized lessons directly from their chosen celebrity. Our software facilitates this process by having celebrities explain complex concepts and summarizing the content of the PDF textbook image. Following this explanation, users will be encouraged to record themselves explaining the concept in their own words. Our software then analyzes the PDF, generating keywords that reflect the user's understanding of the concept. This feedback loop empowers users to gauge their comprehension levels and identify areas for improvement. Additionally, our software compares the keywords generated to the user's recorded explanation, identifying any missed words or concepts. This comprehensive analysis is then provided to the user, allowing them to see exactly where they can improve their understanding.

How we built it

We built CelebLearn on react with typescript. For the lip sync portion of the software, we used Sync Labs Lip Sync API and to connect the front end with the backend, we used FastAPI. Lastly, we utilized OpenAI to summarize information that the user provides with their uploaded PDF. It is also used to generate the transcript of the video and keywords that are used to test the user’s knowledge of a certain area.

Challenges we ran into

This project involved many separate API calls and piecing together several consecutive parts to make the functional program. It made the connection between the front and back end difficult to manage. One of the most difficult parts of generating the DeepFake, especially given the limited number of APIs and resources available online — we needed to compromise. We had to run each step separately (get template audio → change voice of audio → take template video → add lip syncing and combine the two), which was hard.

Accomplishments that we're proud of

We are proud that we managed to create such intricate and complex software in such little time. Specifically, we have never used ML models to create deepfakes as the concept is still relatively new to us. Additionally, it was the first time for three of our teammates to create a React website.

What we learned

Throughout the hackathon, we participated in many workshops and created many connections. We engaged in many conversations that involved certain bugs and issues that others were having and learned from their experience using JavaScript and React. Additionally, throughout the workshops, we learned about the importance of incorporating accessibility features in coding software, which made us understand its cruciality.

What's next for CelebLearn

CelebLearn strives to continue its services for teaching people to learn educational concepts in a fun and enticing way. For future use, we plan to implement a feedback box for our users to communicate with us about problems with our program so that we can work to fix them. We hope to add more celebrity options and even allow users to generate their own by inputting which individual they would like to see on the software.