Open Context

A platform for open-source course data collection, and quiz generation

Comment

Inspiration

Access to course resources is often fragmented and lacks personalization, making it difficult for students to optimize their learning experiences. When students use Large Language Models (LLMs) for academic insights, they often encounter limitations due to LLMs’ inability to interpret various data formats—like lecture audio or photos of notes. Additionally, context often gets lost between sessions, leading to fragmented study experiences.

We created OpenContext to offer a comprehensive solution that enables students to manage, customize, and retain context across multiple learning resources.

What it does

OpenContext is an all-in-one platform designed to provide students with personalized, contextually-aware resources using LLMs. It aims to make course data open source by allowing students to share their course materials.

Features:

  • Upload and Process Documents: Users can upload or record audio, PDF, and image files related to their classes.
  • Chat Assistant: Users can chat with an assistant which will have the context of all the uploaded documents, and will be able to refer to course materials on any given questions.
  • Real-Time Audio Transcription: Record lectures directly in the browser, and the audio is transcribed and processed in real time.
  • Document Merging and Quiz Generation: Users can combine documents from different formats and generate quizzes that mimic Quizlet-style flashcards.
  • Progress Tracking: After completing quizzes, users receive a detailed summary of their performance.

Full User Flow: The user faces a landing page where they are prompted with three options: Create a new chat, Upload documents, Generate quizzes. As the user navigates to Upload documents, they have the option to record their current lecture real time from their browser. Or, they can upload documents ranging from audio, pdf and image. We are using Tesseract for Optical Character Recognition on image files, OpenAI’s SpeechtoText API on audio files, and our own PDF parser for other class documents. The user can also record the lecture in real time which will be transcribed and processed in real time. After the transcription of a lecture or another class document is finished, it is displayed to the user. They will be able to create a new chat and ask our AI assistant anything related to their course materials. The assistant will have the full context of uploaded documents and will be able to answer with references to those documents. The user will also have the option to generate a quiz based on the transcription of the lecture that just got recorded. They will also be able to merge multiple class documents and generate a custom quiz out of all. The quizzes will have the format of quizlet flashcards, where a question is asked, 4 answers are provided as options and after an option is chosen, the website will prompt with either the chosen answer is correct or incorrect. The score for each question is calculated and at the end of the quiz a summary of the performance is written for the student user to track their progress.

How We Built It

  • Frontend: Built with React for a responsive and dynamic user interface.
  • Backend: Developed with FastAPI, handling various tasks from file processing to vector database interactions.
  • AI Integration: Utilized OpenAI's Whisper for real-time speech-to-text transcription and embedding functionalities.
  • OCR: Tesseract is used for Optical Character Recognition on uploaded images, allowing us to convert handwritten or printed text into machine-readable text. Infrastructure: Hosted with Defang for production-grade API management, alongside Cloudflare for data operations and performance optimization.

Tech Stack:

  • Figma: We used Figma to design our warm color palette and simplistic theme for pages and logo.
  • Python: Python scripts are used ubiquitously in OpenContext whether it’s for our deployment scripts for Defang or pipelines to our Vector Databases, our API servers, OpenAI wrappers or other miscellaneous tasks, we utilize Python.
  • React: React framework is used to build the entire components, pages, and routes on the frontend.
  • Tesseract OCR: For converting images to text.
  • FastAPI: We have multiple FastAPI apps that we use for multiple purposes. Having endpoints to respond to different types of file requests from the user, making connections to our Vector DB, and other scripting tasks are all handled by the FastAPI endpoints that we have built.
  • OpenAI API: We are using multiple services of OpenAI such as the Whisper for ASR or Text Embeddings for later function calling and vector storing.

Sponsor Products:

  • Defang: We used Defang in order to test and host both of our API systems in a production environment. Here is the production PDF API: https://alpnix-pdf-to-searchable--8080.prod1b.defang.dev/docs
  • Terraform: We used Terraform, a main.tf script, to test and validate our configuration for our deployment services such as our API hosting with Defang and nginx settings.
  • Midnight: For open-source data sharing, Midnight provides us the perfect tool to encrypt the shared information. We created our own wallet as the host server and each user is able to create their own private wallets to share files securely.
  • Cloudflare: We are using multiple services of Cloudflare…
    • Vectorize: In addition to using Pinecone, we have fully utilized Vectorize, Cloudflare’s Vector DB, at a high level.
    • Cloudflare Registrar: We are using Cloudflare’s domain registration to buy our domain.
    • Proxy Traffic: We are using Cloudflare’s proxy traffic service to handle requests in a secure and efficient manner.
    • Site Analytics: Cloudflare’s data analytics tool to help us analyze the traffic as the site is launched.
  • Databricks: We have fully utilized Databricks Starter Application to familiarize ourselves with efficient open source data sharing feature of our product. After running some tests, we also decided to integrate LangChain in the future to enhance the context-aware nature of our system.

Challenges we ran into

One significant challenge was efficiently extracting text from images. This required converting images to PDFs, running OCR to overlay text onto the original document, and accurately placing the text for quiz generation. Ensuring real-time transcription accuracy and managing the processing load on our servers were also challenging.

Accomplishments that we're proud of

  • Tool Mastery: In a short time, we learned and successfully implemented production environment tools like NGINX, Pinecone, and Terraform.
  • API Integration: Seamlessly integrated OpenAI’s Whisper and Tesseract for multi-format document processing, enhancing the utility of LLMs for students.
  • Quiz Generation Pipeline: Developed an efficient pipeline for custom quiz generation from multiple class resources.

What we learned

  • Infrastructure Management: Gained experience using Defang, Terraform, and Midnight to host and manage a robust data application.
  • Prompt Engineering: Through David Malan’s session, we enhanced our ability to prompt engineer, configuring ChatGPT’s API to fulfill specific roles and restrictions effectively.

What's next for Open Context

We aim to develop a secure information-sharing system within the platform, enabling students to share their study materials safely and privately with their peers. Additionally, we plan to introduce collaborative study sessions where students can work together on quizzes and share real-time notes. This could involve shared document editing and group quiz sessions to enhance the sense of open source.

Built With

Share this project:

Updates