LookAhead

Inspiration

LookAhead's goal is to enhance the lives of visually impaired people by improving and simplifying their daily experiences. At Hack the North 2023, we were also granted a unique opportunity to work with AdHawk's powerful eye-tracking technology for free. Recognizing this as a special opportunity with potential to help many people, we also wanted to integrate this feature into our project, which is ultimately what led us to our topic.

What it does

LookAhead supports the visually impaired through cutting-edge eye-tracking technology glasses. It has two main features, which are object identification and description and obstacle warnings. Users can easily switch between these three modes and interact with the software through Iris, our built-in voice-activated assistant!

How we built it

The main technology that we use is AdHawk's Mind-Link Glasses, which records through a small camera on the glasses' frame and can detect eye / pupil movements. Additionally, we used several Python libraries to handle video processing and AI analysis on the software side, including OpenCV, Google Cloud Vision API, SpeechRecognition, and PyAudio.

Challenges we ran into

The biggest technical challenge that we faced, which took a large portion of our time to resolve, was figuring out what the statistics that AdHawk provides meant (in particular the physical coordinates). Being a niche technology with limited documentations, our questions were not simply answered online, and we became more and more concerned about how to use the data properly. In the end, we brought our questions to the AdHawk's sponsoring booth themselves. With their help, we were finally able to calibrate the glasses and optimize our accuracies.

Accomplishments that we're proud of

Firstly, we are proud as a team for mastering this high-level, specialized technology in such a short period of time. Our chosen topic emerged from team consensus, so each of us had genuine interest in learning the hardware and libraries for this project. Secondly, we each take our individual pride in having the motivation to power through, especially during the 2nd night when we barely slept. Even if we were tempted by a good night's sleep, having the presence of the other enduring group members around helped us pour in our best work until the sun rose.

What we learned

Given the experimental and trial-and-error nature of our project, a lot of scientific research had to be done to collect necessary variables to understand patterns, relations and bounds. We took this as an experience to practice our scientific skills, in organizing data and conducting several experiments with adjusted parameters efficiently.

In terms of hard-skills, this was definitely a learning experience for all of us to work with this type of hardware. It is likely that we will not have the chance to use this amazing technology for free again in the future, which is why we cherished this opportunity and pushed our learning to its limits (by that, I mean we goofed around a lot and did crazy things with the tech).

What's next for LookAhead

One of the charms of our project is its scalability. Adding new features is simple, since the baseline of eye-tracking and image recognition - the hardest part - is already done. Some new features that we plan to add in the future are text-reading modes, caution with stairs, facial emotion detecting, target object locator, and many more ideas. With many promising modern advancements and leading companies conducting research in the field, the outlook is bright for eye-tracking technology in the near future.