MemoAR

Inspiration

"Remember Me"

That one scene in Coco—when Miguel sings to Mama Coco, and her eyes light up as memories rush back—reminds us that memory is more than information. It’s identity. It’s connection. And when dementia takes that away, it doesn’t just affect the person—it affects the whole family.

At LA Hacks, we asked: What if technology could be Miguel’s guitar? What if it could help someone remember—faces, names, places, even joy?

Introducing MemoAR—an AR-powered assistant designed for people living with dementia.

What it does

MemoAR is an augmented reality system that helps people with dementia recall and relive their memories. Using Snapchat Spectacles, users can trigger “memory capsules” with a simple gesture, bringing up photos, videos, audio, or even songs anchored to real-world faces and places. Our AI agent, Snapy, is always ready to answer questions, describe the environment, and provide conversational support. The web portal allows families to upload memories, train the face recognition model, and review chat history.

How we built it

We built MemoAR in two parts:

AR Lens: Developed in Snapchat Lens Studio using JavaScript/TypeScript, with a custom computer vision model (TensorFlow Lite) trained on just 5 photos per person for real-time face recognition. Snapy, our AI agent, is powered by Google Gemini 2.5 Pro.

Web Portal: Built with React and Tailwind CSS, the portal lets users upload memories and connections, stores everything in MongoDB Atlas, and provides a chat history interface. The backend is powered by FastAPI and Python.

Challenges we ran into

The gap between our vision and what’s possible with current tools was a major challenge. Implementing advanced AR features in Lens Studio was tough, especially as first-time users. We faced issues like limited computing power, lack of APIs, and sparse documentation. Even simple features like text-to-speech required us to test multiple providers: OpenAI TTS, Gemini (which we try but found out cannot use api for TTS), Groq TTS(limited), and Eleven Labs (which had credit limits). Despite these hurdles, we found creative solutions and kept moving forward.

Accomplishments that we're proud of

We’re especially proud of training a computer vision model with just 5 photos per person and achieving real-time, accurate face recognition in AR. Seeing the system correctly identify everyone from the trained data was a huge milestone and a testament to our team’s persistence and innovation.

What we learned

We learned how to build AR applications in Lens Studio from scratch, and the experience was both challenging and rewarding. We also gained deep insights into using the Gemini API to deliver highly personalized, context-aware responses for users with dementia. This project pushed us to explore new technologies and think creatively about user experience.

What's next for MemoAR

We plan to integrate MemoAR with calendar and medical apps to provide even more value to users. We also want to make it easy for users to add photos and descriptions of important events, so they can always remember the moments that matter most. Our journey with MemoAR is just beginning, and we’re excited to see where it leads!