ClassX

No more boring lectures thanks to Vision Pro + AI video

Comment

ClassX logo

ClassX = Vision Pro + Classroom RAG + Generative 3b1b GIFs

  • Fuse online + offline classes to ‘ultralearn’ (aka cram) class quickly!
  • Best-in-class multi-query RAG (retrieval augmented generation) shows relevant TA notes, LaTeX equations, YouTube videos, and 3b1b videos.
  • Whisper audio transcript can rewind to older transcript entires
  • [Generative Video] Generate 3blue1brown AI videos with GPT-4 and Manim rendering engine!!

Exploring the intersection of Mixed Reality, Spatial Computing, and Large Language Models in education. The integration of Vision Pro and AI technologies offers a unique opportunity to enhance learning experiences through immersive, interactive content. Our project, ClassX, is designed to leverage these advancements, focusing on the practical application within an educational context.

Our mission with ClassX was to develop a VisionOS app that enriches learning by offering a multi-dimensional platform for students to engage with educational materials. Recognizing the potential for broad application across various fields, we chose to concentrate on education, aiming to provide a solution that addresses the needs of modern learners.

ClassX is a virtual learning environment where students can interact with an array of educational resources, including videos, PDFs, and LaTeX-rendered documents, all presented within a virtual lecture hall setting.

Features of ClassX

  • Dynamic Educational Material Display: ClassX allows users to navigate through and interact with a variety of media types, all integrated seamlessly into the user’s visual field.
  • AI-Driven Content Personalization: We use TogetherAI and Mistral Mixture of Experts to customize the learning experience. We adapt the lecturer's style into a form that fits individual learning styles, preferences, and progress.
  • Engaging Learning Methods: Through interactive elements, such as quizzes and exercises, ClassX transforms traditional learning materials into engaging, interactive experiences.
  • Comprehensive Academic Integration: ClassX offers direct access to a wide range of academic resources, employing sophisticated search technologies to provide relevant, up-to-date materials.

Development Process

The development of ClassX was characterized by a collaborative and methodical approach, emphasizing efficiency and technical innovation. The team utilized a range of development tools and platforms, including Xcode, OpenAI API, Mistral, and LaTeX libraries, to create a robust educational platform.

Stack

Server

  • Together AI: Mistral 7x8b Mixture of Experts chat model
  • OpenAI: text-embedding-ada-3 embedding
  • Chroma multi query vector search. Each document and transcript maps to many keys, and Chromadb reranks n->n SQL mapping by similarity.
  • 3b1b Manim: Grant Sanderson’s Python math rendering engine. GPT-4 generates 2d animation scenes as executable code, creating 10-sec crystal-clear AI animated video (without OpenAI Sora 😉).
  • FastAPI: serves generated video and APIs

visionOS App

  • 100% Swift and SwiftUI: fully native app!
    • handles animations, images, webviews, and more
    • native visionOS dynamic layout grids and resizing support without breakpoints
  • Whisper (Local): Transcribe audio offline with timestamps
    • Live streaming via AVFoundation
  • LaTeX renderer (with regex to extract LaTeX sections and handle inlining)

The ClassX Experience

  • Virtual Lecture Halls: ClassX adds a virtual environment on top of an existing one. It enhances/supplements the boring lecture hall experience with educational content to facilitate a comprehensive and engaging learning experience.
  • Customized Learning Journeys: AI technology assesses each learner's unique profile to deliver personalized content, optimizing the educational experience.
  • Interactive Learning Tools: ClassX enhances learning retention through interactive quizzes and exercises, providing instant feedback to reinforce understanding.

Challenges and Achievements

  • Developing for Vision Pro: Tailoring ClassX to the innovative capabilities of Vision Pro required creative problem-solving and technical acumen.
  • Complex Content Integration: The integration of LaTeX into mixed reality posed significant challenges, but the team successfully achieved smooth rendering of intricate academic content.

Future Directions

Moving forward, ClassX aims to expand its content offerings, integrate live tutoring capabilities, and explore the potential of augmented reality (AR) for practical learning applications.

Considerations

While ClassX represents a significant step forward in educational technology, we are conscious of the challenges ahead, including device accessibility and data privacy concerns.

Thank you for exploring ClassX. Our work represents a commitment to advancing educational technology for a brighter, more informed future.

Built With

Share this project:

Updates