ImmersiveBook2.0

Inspiration

Ever dreamed of diving into your favorite books? With ImmersiveBook2.0, any book turns into an immersive VR visual book!

What it does

Books, novels, or any texts you input are turned into immersive 3D VR space that you can gaze around or walk into with Meta Quest3 or any other VR devices of your choice.

How we built it

three.js + WebXR guarantee a cross-platform experience just with a web browser, without locking into a specific VR device and development environment.

Accomplishments that we're proud of

• < 1 min end-to-end to generate an immersive 3D VR space
• Highly scalable, any texts/books can be converted with minimum fine-tuning/post-processing.
• Light-weighted hosting with WebXR without needing a GPU server/PC to render (just on Meta Quest 3).

Challenges we ran into

• 360-degree point cloud generation takes too long and is too heavy to render the point cloud on a VR device.

What we learned

• Collaboration along different expertise (frontend, ML, 3D art, etc)
• Quick experimentation and implementation
• Potentials and limitations of AR/VR

What's next for ImmersiveBook2.0

• Audiobook + animation + VFX to make it further immersive
• Gamified/interactive/conversation content to entertain users
• Full 360-degree FoV coverage (by point clouds and Sky Box/Dome)
• Improve the visual quality by 3D Gaussian Splatting, higher poly/vertices counts, and procedural 3D mesh/asset placement.

Built With

• amazon-web-services
• dust3r
• gpt-4o
• node.js
• python
• quest3
• runpod
• stablediffusion
• three.js
• webxr