MemoryMake

Turn your 2D images—or even your words—into immersive 3D memories

Comment

Inspiration ✨

As memories pile up in our camera rolls or remain abstract ideas, we wanted to create a way to truly experience them. MemoryMake was born from the idea of transforming ordinary photos and text prompts into dynamic, navigable, and visually engaging 3D environments—so users can explore their cherished moments instead of merely scrolling past them. In a society where everyone is constantly thinking about going forwards, we wanted an opportunity to appreciate our memories and live in them once again.

What it does 📸➡️🖼➡️3D

  • Upload or Prompt ⬆️
    Provide a photo or type a quick prompt describing the scene you envision.

  • Style Selection 🎨
    Choose from artistic styles like Photorealistic, Monet, or Van Gogh, which will overlay onto a realistic place that you upload

  • 3D Model Generation 🏗️
    Our backend uses MiDaS depth estimation combined with Open3D mesh generation, and our own algorithm to turn point-clouds to cylindrical coordinates to generate a .obj file—complete with depth data for realistic perspective.

  • Immersive Exploration 🚀
    The 3D model is returned to the frontend, where you can zoom, pan, and navigate around your “memory.”

  • View Past Creations ⏮️
    Seamlessly browse through all your previously generated 3D memories.

Whether it’s a snapshot from your last vacation or a scene you’ve only imagined, MemoryMake makes each memory come alive in three dimensions.

How we built it 🛠️

Backend

-FastAPI⚡ Handles image or text inputs, orchestrates style selection, and calls MiDaS for depth mapping.

-MiDaS🌐 Generates the depth data needed to produce accurate .obj models.

-Open3D 🧩 Creates 3D meshes from the depth data and point clouds. Using our custom algorithm, we extrapolate finer details from the point cloud to enhance the accuracy and realism of the generated models.

-Neural Style Transfer (NST) 🎨 Applies artistic styles to the 3D renderings by overlaying chosen artistic elements, ensuring the final output is both visually striking and thematically aligned with user preferences.

Frontend

  • React 💻
    Creates an intuitive UI and interactive environment where users can explore their 3D memories.

  • 3D Rendering 🖥️
    Displays the .obj files, allowing zoom, navigation, and style previews.

Pipeline

  1. User selects style and uploads a photo or enters text.
  2. FastAPI receives the input, invokes MiDaS for depth estimation, and returns the 3D model.
  3. The frontend renders the model, enabling interactive exploration and storing past creations.

Challenges we ran into 🚧

  • Depth Accuracy ⚖️
    One of the major challenges we faced was developing the algorithm from a MiDaS generated depth image to a realistic panorama. We had to consider ways to turn a 2D image into cylindrical coordinates that could not only wrap around the viewer, but also highlight the extruding features of the image for full immersion. A lot of mathematical analysis also went into the effort.

  • Prompt to Image Prompting was much harder than we expected, because we had to investigate into really detailed prompting to allow the diffusion model to generate a panorama photo that would render realistically through the MiDaS to Open3d pipeline.

  • Style Integration 🎨
    Blending artistic style choices was challenging because we experimented with various models before coming to one that satisfied us, which was the Neural Transfer Styles.

  • Performance & File Handling ⚙️
    Managing .obj models efficiently to keep interactions smooth on the frontend. .obj files are very large so it is hard to handle them in the frontend

  • Seamless User Experience 👥
    Making a 3D environment feel natural and intuitive, especially for users new to immersive content. It was our first time using three.js, so we experimented a lot with the features.

Accomplishments that we're proud of 🏆

  • Depth Accuracy ⚖️ Successfully developed a custom algorithm to convert MiDaS-generated depth images into realistic panoramas. This included mapping 2D images into cylindrical coordinates to wrap around the viewer and accurately highlight extruding features for full immersion, backed by extensive mathematical analysis.

  • Prompt to Image 🖼️ Mastered detailed prompting techniques for diffusion models, enabling the generation of panorama photos that rendered realistically through the MiDaS-to-Open3D pipeline, ensuring high-quality outputs.

  • Style Integration🎨 Achieved seamless blending of artistic style choices by experimenting with multiple models and ultimately adopting Neural Style Transfer for consistent and visually appealing results.

  • Seamless User Experience 👥 Designed an intuitive and natural 3D environment for users new to immersive content by leveraging and experimenting with Three.js, successfully implementing features that enhanced interactivity and engagement.

What we learned 📚

  • Depth Accuracy ⚖️ We are now very familiar with the open3d library, and we also learned how 3d rendering is generated through calculations.

  • Style Integration 🎨 Gained insight into blending artistic styles effectively by experimenting with various models, ultimately learning the strengths and limitations of Neural Style Transfer for consistent and high-quality results.

-Performance & File Handling ⚙️ Learned strategies to handle large .obj files efficiently, optimizing performance on the frontend while maintaining smooth and responsive interactions for users.

What's next for MemoryMake 🚀

  • Fully immersive We want to expand this idea into the VR world. We already have the algorithms necessary!

Built With

Share this project:

Updates