Handwrite-AR

Inspiration

Around 1 in 10 people have Dyslexia which is a learning difficulty which causes problems with reading, writing and spelling. This can severely limit the possibilities of so many young people in schools. There are many tools to help with this condition however these limit the freedom of these people by preventing them from writing easily with pen on paper.

What it does

Handwrite-AR is a real-time, augmented reality, accessibility tool which provides a live overlay on handwritten text. This checks spelling, punctuation, and can translate text. In addition, it provides a narrator and definitions of handwritten words. As our product is aimed at people with Dyslexia, it also has various colour filters for an infinitely better reading experience.

How we built it

Handwrite-AR is built using C# and various APIs. It takes a base-64 image as a required input. It then uses Google's Optical Character Recognition (OCR) API to detect the handwriting and parse this into a list of words. These words are then run through Microsoft Azure's Spellchecker API where potential errors are flagged, with a list of possible corrections. Optionally, the user can translate the text as well, which will then run the text through Google's Translate API to accurately translate the text into a specified language. This processed data is passed back to the client where it highlights incorrect words in real-time and lists the possible corrections adjacent to the incorrect word.

In addition, the user can select words in the AR environment to show a dictionary definition. In the interface, there are also options to translate the text to different languages and to apply colour filters to aid with reading.

Challenges we ran into

Using a number of different APIs, which rely on one another, meant that we can get high latency when processing the data. This gives the impression of a slow and delayed application. We overcame this issue by hosting some of the APIs on the same machine as the backend. This reduced our latency and decreased the round trip time making the AR application smoother and more responsive.

Another issue we had was the VR headset we were provided with to test our application on was not in developer mode. This was required to quickly deploy and test our AR environment to. We had to share resources with other groups making the AR development slow.

Accomplishments that we're proud of

We had a proof of context prototype working very quickly with the OCR and Spellchecking APIs. This gave us the confidence to build around this and we were able to expand our product a lot.

What we learned

None of us were too familiar with C# so building a big project in this has been a great learning experience.

What's next for Handwrite-AR

As it is a working product, it can be used in the real-world. One of our members has Dyslexia and would consider using it to help his writing. It can be enhanced slightly on the frontend to make the user experience better.