Rosetta Glass

Inspiration

Whether it be recreational reading, academic research, or even reading to learn a new language - there's all been times where we’ve wished we could just read faster, understand better, and retain information for longer. Introducing: the Rosetta Glass.

What it does

The Rosetta Glass is an ultra-intelligent reading companion that will identify when the user is having difficulty with a particular passage, word, or sentence using Ad Hawk's eye-tracking glasses and suggest alternative explanations with Cohere's NLP capabilities. The Rosetta Glass also doubles as a translation aide when reading foreign languages, eliminating the clunkiness of having to go back and forth between Google Translate and your favourite foreign novel. At the end of the day, both the text simplification and translation modes record the user's problematic texts into our CockroachDB database for easy reference through our handy iOS app.

How we built it

Our main, desktop application to track the user’s eye movements, detect when the user is struggling with a passage, and display text simplifications / translations is built using Python with the Qt framework. The app leverages Google Cloud Vision AI to identify coordinates for each word on the screen, so when we detect confusion from the user (for example, if their eyes are fixated on an unfamiliar word), we cross-reference the location of the user’s gaze with the coordinates of each word to determine which word(s) to simplify / translate.

After simplifying / translating the text, the desktop app POSTs the original text along with the processed text to our AWS Lambda backend through HTTP API. The backend is responsible for storing these entries into our trusty CockroachDB serverless database.

Finally, we developed an iOS app using …. which fetches the user’s problematic texts from the past week using GET requests to our backend.

Challenges we ran into

This was our first time working with the Python Qt library and we ran into several issues with rendering our sidebar that swallowed hours and hours of our time. Because our hack also included a hardware project we had to account for things like blinking, calibrating the glasses and offset eye tracking. Many of these procedures needed their own algorithms and took a lot of debugging and discussions to build.

The software for the glasses was also intensive and only one teammate had a computer that was powerful enough and had the right operating system to run it but with the API calls and the software, the CPU load became very high and the laptop started freezing.

We had challenges with almost everything we tried but they all paled in comparison to these two.

Accomplishments that we're proud of

This was a pretty complex project with lots of different technologies that were brand new to us (CockroachDB, AdHawk MindLink, Cohere, and several GCP APIs).

Although, towards the end we were in integration hell, we are still proud of the fact that we got each component working atleast independently.

Despite all the challenges we faced, we’re proud of what we were able to do in this limited time span.