Assisted Reader

Wearable tech addon for the visually impaired to read text using OCR

Comment

Inspiration

Many visually impaired people have difficulties on a day-to-day basis reading text and other items, and other comparable technologies and software are either expensive or bulky and impractical to use. We sought to create a cheap, light and reliable system for text reading.

What it does

A wearable that reads aloud text to its user through a mobile app. It was made for the visually impaired who have trouble reading text that doesn't have a braille translation.

How we built it

We used an ESP-32 Cam, mounted in a 3d printed enclosure atop a hat. The camera hosts a WiFi Local Access Point, to which the user's smartphone is connected to. The smartphone processes the image with an OCR, and sends the resulting text to a text-to-speech API, which is played to the user.

Challenges we ran into

  • We forgot to bring a charger and HDMI cable for our raspberry PI and were unable to rely on its processing power.
  • We tried to run ML models on the ESP32-CAM but we were only able to small and simple models due to the hardware limitations.
  • We were unable to send images over Bluetooth LE because of the low data transfer limit and we were unable to connect the ESP32-CAM so we opted to create a soft access point on the esp32 so other devices could connect to it to retrieve images.
  • Getting a 3D profile of the hat (we borrowed stole from xero) was challenging because we couldn't measure the spherical shape of the hat.
  • We had a lot of trouble retrieving images on our React Native app, making us switch to React. After a lot of trial and error, we finally got the ESP32-CAM to stream images on a ReactJS website, but we didn't have enough time to combine the text recognition with it.

Accomplishments that we're proud of

  • 3d Printed Shroud
  • Building a website and implemented Machine Learning model
  • Streaming ESP32-CAM to our website

What we learned

  • Always allow more overhead when possible to allow for changes while creating your project
  • Create a schedule to predict what you will have time for and when it should be done by
    • ML training is very time-consuming and it is very likely that you will have bugs to work out
    • Start with a smaller project and work up towards a larger and more complete one
  • New things our team members have experienced: React (JS & Native), ESP32-CAM, Tesseract (js & py), NodeJS, 3d modeling/printing under a time constraint, getting free food

What's next for Assisted Reader

  • Fix issues / merge the ESP32-CAM with the ML models
  • Better form factor (smaller battery, case, lower power usage)
  • Use automatic spelling correction to ensure Text-to-Speech always reads proper English words
  • More ML training for an improved OCR model
  • Translation to other languages for a larger customer base
  • Cleaner and more modern User Interface
  • Add support for Bluetooth and connecting to other Wifi networks

Built With

Share this project:

Updates