Inspiration
We were fascinated by AR technology and wanted to build a intuitive AR software that could solve real world problems. After thinking about a problem that we could solve, we were hopeful to help people learn English. 21% of adults in the US are considered illiterate and we were hoping to help solve that problem by creating a system to help them learn to write.
What it does
Our python program allows users to overlay a stencil of their choice on a piece of paper in real time with the use of augmented reality. Our goal is to help people learn to write, so one of the main ways you can upload a stencil is by inserting text. Users can use the speech button for those who do not know how to type. Additionally we used an API to convert other languages to English for those who speak a foreign language. Lastly, you can upload an image that works as a stencil. In the Camera Scene, the text-converted-to-image stencil or the uploaded image would be placed onto a page that the camera detects using an analysis algorithm. Pointers can be used to help specify bounds. The user would proceed to trace over the text/image to practice writing/drawing.
How we built it
We utilized a variety of opencv filters and contours to detect a piece of paper in the camera. We then used a homography matrix to project the stencil onto the paper. The flask backend would communicate the camera feed with the react native front end where the app UI was built, but there were some bugs in this process that led to issues. The camera would analyze the feed, detect the page, and then overlay the image after computing the location and the distortion needed on the stencil image.
Challenges we ran into
Detecting the paper without the use of deep learning image segmentation was difficult. Especially when the paper was covered by a hand or pencil. Additionally, there was a host of issues that came into play about connecting Flask and React since we wanted to transmit a live video stream at all times from both ends. In the end, we attempted to modify a single image dynamically, but even this had its issues.
Accomplishments that we're proud of
We are proud of obtaining a system that is capable of detecting pages and accounting for the change in perspective on the projection of the stencil on the paper.
What we learned
We learned about image detection and AR Technologies as well as grasping a more in depth understanding of Flask, React, and opencv.
What's next for Write Right
We would be fine-tuning the app and fixing the connection issues between the backend and frontend servers. Additionally, we had ideas of allowing the user to scan text that would be spoken out since a portion of the users would not be able to read.
Log in or sign up for Devpost to join the conversation.