ThirdEye.py

Bringing a new perspective to the functionally blind

Comment 1

Inspiration

We were inspired to do this by the JP Morgan chase Accessibility challenge. We wanted to make a product that could allow the functionally blind to go about their lives with more independence. We were brainstorming and we remembered that many blind individuals use the internet with Text-To-Speech solutions, and we used that as a starting point for our solution that turns the outside visual environment, into audio stimuli.

What it does

We use a webcam and a pair of ultrasonic sensors in tandem with the Google Cloud Vision API and raspberry pi to build a standalone unit for the purpose of object and person identification with distance sensing and OCR capabilities. To effectively communicate with the user, we use tactile momentary switches, as well as, Google Text to Speech API through the raspberry pi and command line mp3 player.

How I built it

The hardware for the system is comprised of a 720p webcam, two HCSRO4 ultrasonic sensors, a raspberry pi 4B, and a portable power bank. Internet connectivity is necessary in order to access the API's we use and so wifi / hotspot is used/planned for a consumer grade project. A breadboard is included in this in order to hold the momentary switches as well as the necessary electronics, but we plan for a a custom PCB if the project is to be continued.

For software we used Google's API to detect objects and persons to appended it to an array. In a separate function we set up OCR functionality, and set up GPIO access to cycle between the two functions, as well as, toggling automatic distance detection to alert the user. All of the info is then processed and then sent to another Google API to enable us to "talk" to the user.

Challenges I ran into

Among the problems we encountered I think the most time consuming was reformatting the raspberry pi and getting it in a programmable state. The problem was each format took around 20 min and we managed to accidentally brick the OS a couple times. But eventually we were able to get it into a programmable state and it was smooth sailing from then on. Hardware wise we faced challenges in prototyping time. Because we used a 3D printer for a lot of the mounting parts on our project we had to be sure that our print would finish fast and be fully functional. To do this we double checked our measurements and had multiple people look over the models before we sent them to the printer.

Accomplishments that I'm proud of

We are proud of our ability to quickly determine what our project would be and execute on it. We are extremely pleased with our completely working prototype, despite having little to no knowledge on the items used. We are proud of our ability to quickly troubleshoot problems that arose from running interpreted language on an embedded system. And most of all we are extremely proud of the work ethic and results we achieved from these 24 hours.

What I learned

During this Tamuhack our team learned how to use google cloud vision API, how to develop code for use on a raspberry pi, quickly and effectively brainstorm and come up with a design, and finally how to collaborate in 3d modeling an effective prototype.

What's next for ThirdEye.py

If we were to continue this project we would probably work first on a miniaturized product with better packing and smaller ultrasonic sensors. Next, we would make a smaller custom PCB rather than Breadboard for user controls, and add textured buttons for easier differentiation. We would love to add a dedicated camera, as well as, SIM Functionality so that it doesn't have to be tethered to another device.

Built With

Share this project:

Updates