-
screenshot of the model predicting the handwritten images
-
-
screenshot of the program running predictions and testing the given data
Inspiration
Studying and working in the field, writing is considered by many an essential part of life. However, for some, it is considered cumbersome, with many people suffering from dysgraphia, a neurological writing disability. However, dysgraphia is not untreatable. With motor training, it can be treated, with improvements to handwriting. That is why we created ScriptSense, an AI program designed to detect handwriting via OCR and image detection.
What it does
This AI tool analyzes handwritten data and then interprets it into a more legible print format. People with dysgraphia have a neurological disability that inhibits their writing capabilities and this tool helps with that. It works by using a published data set from Kaggle. The data set was then used to deeply train the system in detecting what was written. The original data set however, comprised of well-written word snippets however. What we did was supply the model with more information from less legible font and try to see if it could predict these characters.
How we built it
We built this tool with the provided data set and a few additional sets of our own. With the power of python and the provided extensive library for OCR such as OpenCV, Matplot, and NumPy, we worked to train and extract best results from the tool. In essence, the tool worked by taking an image we wrote on a whiteboard and reading it in image = cv2.imread("o.jpg"). Once read, the image is reshaped for a reduced, but efficient computation. The image is submitted and then processed by the model which makes a prediction on what the image is. This prediction is a NumPy array of "matches" according to a dictionary where 0 maps to A and 25 maps to Z. The model's prediction was the highest value in the array and it was a matter of finding what value that is. With the power of np.argmax, the highest value was found and then was compared with what the expected value was supposed to be
Challenges we ran into
We ran into a lot of mishap. One of the primary issues we faced was inconsistent results. Our first try yielded a correct prediction, however, with additional tries, we began to notice that the model was yielding a different result every time. We initially thought the problem was not running enough reruns with more epochs. While that was a temporary solution, with only marginally increasing the consistency of our results, we started to read through the model more thoroughly, and we began to properly comprehend the specific parameters of the data we needed to feed in. What we input into the model was 'technically' compatible, but due to poor image preprocessing, the data was essentially faulty, and near-illegible for the model to effectively parse and predict. Thankfully, after fixing the preprocessing parameters, the model yielded significantly more consistent and accurate predictions.
Accomplishments that we're proud of
We're proud of being able to properly deconstruct and properly understand our code, allowing us the depth of understanding to recognize errors (as aforementioned), and be able to improve upon a solid foundation within the limits of our knowledge. We're proud of being able to not only identify, but troubleshoot and repair significant errors in our code, which would have otherwise rendered our project unusable.
What we learned
Within the scope of our given 24 hours, we learnt a lot about TensorFlow, the Keras API, Python, NumPy, OpenCV, and many other Python Libraries. We learnt a great deal about neural networks, how they work, the significance and use of data. It gave us better insight into the powerful and endless capabilities of AI and Machine Learning.
What's next for ScriptSense
Beyond singular letter recognition, we would like to optimize ScriptSense for recognizing strings of characters as words, and eventually fully formed sentences. We would like to also implement a model allowing us to "grade" the input handwriting, and offering solutions to improve them.
Log in or sign up for Devpost to join the conversation.