EyeSpy

Inspiration

Visually impaired people face many challenges in their day to day lives; challenges that often go unnoticed by those with full sight. On average, visually impaired individuals walk 25% slower than their sighted counterparts due to the caution they must exercise while navigating unfamiliar environments. Due to being visually impaired, they receive fewer motion pattern signals in the central nervous system and have lower balance performance, compared with people who have full vision.

Seeing the struggles that visually impaired individuals go through every single day, this prompted us to develop a solution that can improve their quality of life. By providing a tool such as EyeSpy, we aim to reduce the physical constraints.

What it does

EyeSpy uses an iPhone camera to scan the surroundings; it samples images every couple seconds and passes it to a model, the Detectron2 model. The model analyzes the image and performs object detection and creates bounding boxes of the objects near the visually impaired individual. The bounding boxes are processed in the backend and converted into simple english description (i.e there is a backpack on the left). This english description is fed to cohere and the model provides an in-depth description on how to navigate / proceed forward. This description is fed to the individual in an audio format using whisper.

How we built it

We used the Detectron model to do image analysis and object detection. The object detection model generates bounding boxes and we post-process the coordinates of the bounding boxes and the labels to convert the coordinate system into english language. This is fed to cohere to generate in-depth descriptions on how to navigate / proceed forward. The output is converted into audio using openAI's whisper model. And this process is looped through every couple seconds to help the blind individual navigate around.

Challenges we ran into

Initially, we centered our strategy on AdHawk's glasses, leveraging their front-facing camera. Our intention was to execute the entire process using this camera. However, we encountered a setback as the camera was disabled, necessitating an alternative approach to accessing camera data. Additionally, we faced a challenge when processing images with the Mask R-CNN model. Initially, we employed a different model, but it proved unreliable as the generated bounding boxes were highly inaccurate, rendering them unsuitable for obtaining accurate coordinates.

Accomplishments that we're proud of

Deploying an accurate end-to-end model that is able to extract meaningful representations, and then analyze that data using heuristics and Cohere LLMs to drive tangible improvement in the lives of visually impaired people.

What we learned

We learned about a ton of new technologies ie BentoML, Cohere, and TTS (text2speech) models. Outside of the technical aspect o f this solution, we learned about how important team dynamic is when having each team member working on disjoint parts of the solution. We realized that harnessing the strengths of each team member and fostering a collaborative spirit can lead to successful development.

What's next for EyeSpy

Our future vision with EyeSpy is to implement a Speech to Text feature that allows the visually impaired individual to interact with the LLM. This is to create a more seamless experience for the user, enabling a two-way communication between the user and the system.

Built With

• bentoml
• chakraui
• cohere
• elevenlabs
• next.js
• openai(whisper)
• python
• react
• typescript
• vercel