LectureLens VR View

Inspiration

Lectures have never been more accessible, yet learning from them still feels stuck in the past. Students watch long videos, rewind endlessly, and hunt through timestamps just to answer a simple question. We wanted something more natural—more like having a professor right beside you.

LectureLens began as an attempt to make learning active, verbal, and immersive. And with VR, voice input, and RAG, it becomes something more: a learning space where speaking your thoughts is encouraged, not inhibited. Since I am planning on studying abroad in Barcelona, I thought this would be a fun tool to have as I embarked on learning Spanish!

What it does

LectureLens VR View turns any lecture into an interactive, responsive, and VR-friendly learning experience. A user uploads slide PDFs and the associated audio or video recording. The system:

Transcribes the lecture with ElevenLabs

Indexes all content using Gemini File Search

Uses RAG to retrieve only the most relevant passages

Lets users pause playback and speak questions aloud

The result is an AI assistant that answers questions based solely on your lecture materials — like having a personalized chatbot embedded inside the VR lecture space.

How we built it

The frontend is a React (Vite) interface designed to be comfortable inside a VR browser and deployed globally via Vercel. The backend runs on Node + Express (hosted on Render), handling all file uploads, transcription, and RAG orchestration.

Our end-to-end pipeline:

User uploads PDF slides + media

ElevenLabs performs high-accuracy speech-to-text transcription

Transcript is processed and converted into text files

Slides + transcript are indexed into a Gemini File Search store

Gemini 2.5 Flash runs RAG/grounded queries using File Search

Frontend displays instant answers alongside voice-based question input

This architecture gives us a clean separation between VR interface, voice capture, and intelligent retrieval. Buttons are made substantially larger for easy VR viewing.

Challenges we ran into

CORS conflicts between Vercel and Render, especially during voice uploads and API routing

Deployment instability, including multiple active Vercel builds accidentally pointing to the same backend, causing CORS failures and server crashes

Managing a split architecture where the frontend lives on Vercel while the backend runs on Render, each with different networking expectations

Clarifying the product direction — our original idea involved capturing live lectures and generating real-time chatbot answers, but audio streaming + real-time transcription + chunked RAG indexing was infeasible within hackathon time

Ensuring Gemini File Search imported and indexed content correctly, and handling long-running operations without blocking the UI

Accomplishments that we're proud of

A full RAG pipeline with Gemini File Search that actually works end-to-end

A VR-friendly React interface that feels simple and natural to use

Cloud deployment on two platforms (Vercel + Render) w/ GoDaddy Domain Name

High-quality voice-to-text recognition via ElevenLabs

A real “pause-and-ask” lecture system — not a mockup, but a functional prototype

What we learned

How to build with Gemini’s File Search and implement a retrieval-augmented workflow

Practical voice recording + audio upload patterns in React

How to architect a distributed cloud app across separate platforms

How to design UI flows that translate well into VR

The importance of deployment configuration

What's next for LectureLens VR View

Full WebXR integration for a native VR lecture theater

A “Professor AI” mode using system prompts and persona modeling for more instructive answers

Language-learning mode that uses transcripts + voice questions to help users practice speaking and comprehension

Timestamp-aware RAG so answers can jump you to the exact moment in the lecture

Course-level knowledge bases that combine multiple lectures into a unified retrieval system that grows in size with the user

Headset-optimized UI/UX for Meta Quest, Apple Vision Pro, and WebXR-compatible devices

Built With

• css
• elevenlabs
• express.js
• gemini
• javascript
• node.js
• rag
• react
• render
• vercel
• vite