Class Skippers' Comprehension Improver (CSCI)

• 

  Testing with prompts

• 

  Testing with other prompts

• 

• 

  The dog image used to test the app

• 

  The DNA image used to test the app

• 

  The Binary Classification pdf file used to test the app

Inspiration

• Inspiration came from my own experience where I skipped classes to learn by myself and needed to catch up.
• I recognized the need for a tool that helps students catch up with class materials at their own pace while maintaining a high GPA.
• I aimed to leverage AI to summarize and explain class materials, allowing for efficient study sessions.

What it does

• The app is designed to assist students who miss classes by providing an AI-based platform to ask questions and receive concise explanations.
• It integrates text, image, and pdf processing to comprehend educational material uploaded by the user.
• Use cases include generating summaries of documents, extracting Q&A from texts, and explaining concepts depicted in images.
• Intended users are students who need to study class material outside of school hours and educators who wish to provide supplemental learning aids.

How we built it

Frontend:

• Streamlit:
• Allows quick iteration and responsiveness in design, while facilitating the creation of a seamless user interface with minimal code.
• Employs widgets and interactivity features which enable users to upload files, engage with the AI, and receive responses in real-time.
• The app leverages Streamlit's caching to optimize performance, reducing load times for returning users’ desired answers and preserving state across sessions

Backend:

• LangChain:
• Create a conversational AI that can interact with and learn from uploaded materials.
• Integrate text and image processing to create embeddings with FAISS (Facebook AI Similarity Search) for efficient similarity searches and response generation.
• Hugging Face models
• Utilizes pre-trained models from Hugging Face's Model Hub like mistral-7b-instruct-v0.1.Q5_K_M.gguf to analyze the input.
• Incorporate transformer-based architectures to allow the system to understand the context and generate human-like text based on the input materials.

Challenges we ran into

• Faced challenges in designing a user-friendly interface with Streamlit that accommodates the app's complex functionalities.
• Navigated the learning curve associated with LangChain while developing the app.
• Limited access to advanced hardware and financial constraints impacted the performance of the app, leading to longer response times and reduced efficiency in processing large volumes of data.
• Struggled with integrating multiple models due to their lack of compatibility and the limitations of current multi-modal models.
• Documentation gaps and ambiguities led to additional research and self-learning to troubleshoot issues.

Accomplishments that we're proud of

• The app runs successfully. ## What we learned
• Gained extensive knowledge in extracting and synthesizing information from various media types.
• Developed robust problem-solving skills, making extensive use of documentation, community forums, and educational resources.
• Acquired proficiency with LangChain and Streamlit, opening new avenues for creating interesting applications.

What's next for Class Skippers' Comprehension Improver

• Envisions the app evolving into a virtual instructor, capable of interpreting questions across media types and providing detailed, accurate educational support.

Built With

• faiss
• huggingfaceapi
• langchain
• python
• streamlit