Inspiration
While volunteering at the University of Washington Medical Center, I was assigned to the optometry department. Here, I met Raj, a young adult who was diagnosed with retinitis pigmentosa, a rare genetic disorder that narrows vision over time. I didn't have the training to treat him medically, so our interactions were limited to consoling. Over many questions, we were surprised to learn that Raj's biggest emotion wasn't fear due to a lack of safety--it was a loss of self-respect due to feeling helpless. He mentioned wanting to be able to sit in chairs without swinging his cane around, wanting to walk to his mother without her having to come to him, wanting to navigate through more than just his cane. Lumina is our action on the realization that sight isn't what was missing; it was direction.
What it does
Lumina is a smart compass built for blind and visually impaired users. The current minimum viable product is a Raspberry Pi 5 with camera and microphone modules connected to ML models, LLM calls, and Python scripts, with a compass needle that points in the direction the user intends to go. It is intended to give the user more autonomy: "Take me to a chair" - Lumina understands this and uses computer vision and AI-powered scene understanding to guide the user to their chair through a compass that the user can feel for haptic feedback.
How we built it
Workflow of the device:
- Object detection: Mobilenet TFLite models that are filtered with keyword matching (based on the user's input)
- Scene analysis: Gemini 2.5 Flash model hosted on Vercel backend server that the Pi sends requests to when the relevant button is pressed.
- Position estimation: Mobilenet TFLite bounding boxes are given a center point, with the object chosen and saved in short-term memory for that class. The user's position is compared to their desired destination through the camera, and the user is guided through the compass (haptic feedback) and optional voice commands.
- Audio output: LMNT API for empathetic text-to-speech responses.
Challenges we ran into
We realized that visually impaired individuals greatly value their sense of hearing, since that takes a much more major role as their eyesight fades. Considering this, our thoughts and our research made it clear that visually impaired people would rather use their ears for themselves, using another less prevalent sense. This made the compass a clear next choice, with a cane in one hand and a lightweight compass in the other (or attached to the individual's body). Going with the compass meant taking on a significant hardware challenge, especially considering that we only had 3 ampere power cords while the device needed 5 to function properly.
We also faced issues with latency, with a lot of API calls and device-user interactions taking a frustrating amount of time. Our solution to this was to simplify this greatly, only keeping the flows that are essential. This included rethinking the entire flow, rewriting inefficient parts, and processing Machine Learning tasks on the device itself.
Speech recognition was often faulty near the start, with it not catching a lot of what we were saying. We addressed this by adding noise filters, but users still have to speak loudly and clearly.
Accomplishments that we're proud of
- Built a fully functional prototype in 24 hours that a visually impaired person can get real value from. This gives us great hope on how long-term work on this project could benefit the lives of visually impaired folks.
- Integrated several tools that we never used before, including Mobilet, LMNT, and others, in perfect collaboration
- We did a test by blindfolding ourselves, and we succeeded in 8/10 of our tests. This result made us very happy, since our visual impairment was new and unfamiliar, but we were still able to consistently get good results. This proved to us that the technology we built was functional, with a little bit of training.
What we learned
We learned that when you're designing a product to dramatically improve people's lives, it is more important to objectively look at the situation and learn from your users instead of believing that you know what's best. If we had gone with the latter approach, we would have made an audio-based detection system that visually impaired people valued, but not nearly as much as Lumina--the first is like buying insurance, the second is giving a sixth sense.
What's next for Lumina
- Adding gesture-based triggers (hand gestures) would lead to seamless control, allowing users more ways to command the device
- Adapting the hardware so that it can become an attachment to walking canes would make the device much more accessible and handy. This would allow for bigger batteries, greater stabilization, and a seamless product usage.
Log in or sign up for Devpost to join the conversation.