Talk2Me

Home page of Talk2Me web app
Some of the pages from Talk2Me web application
UI example of reading with Talk2Me
Python script result of live audio input and comparison to existing sentence in a book
Overview of Azure incorporation in our project

Inspiration

With Microsoft co-hosting the Hackathon, we decided to explore some Azure features at the beginning of the hack to get inspiration. One feature that initially caught our attention was the Speech API, which allows easy conversion between speech-to-text as well as text-to-speech. As auditory learners, we immediately saw potential for a project which incorporates this cognitive service in an educational platform. 
With this foundation, we were able to develop the Talk2Me idea. As first-generation immigrants who were raised with no family members to practice fundamental English with, we thought it would be a great idea to develop an interactive buddy for beginners to practice language with.

What it does

Talk2Me is a state-of-the-art education platform for beginner language learners. The platform offers a wide array of eBooks that the user can pick from and read aloud. The platform acts as a buddy and listens in as the learner reads the book. The program transcribes the reader's voice in real time and compares each spoken word to the text of the book, identifying words they pronounced incorrectly or are struggling with and showing them the correct pronunciation.  
The platform allows the users to track their own progress and identify books that are suitable for their reading level in the future. The user friendly and straightforward interface allow the users to learn and excel independently.

How we built it

We developed our Front-End with the help of existing website frameworks. We modified the css and javaScript files and coded multiple new HTML pages ourselves to suit our desired website layout and objectives. We also incorporated an interactive audio recorder and saver on our site. This site was deployed through Microsoft Azure as a Web Application. 
Our backend was developed separately. We launched a Cognitive Service resource on Azure and utilized its key to build communication with Azure’s Speech API. With this API, we were able to develop a program that takes in real time input and compares it to an existing text database and identify correctly and incorrectly pronounced words.

Challenges we ran into

Our main challenge was finding a method of integrating our front and back-ends together. Although we were able the individual parts launched and working, it was difficult to incorporate the two in order to get real time input & processing done over the web or on the site.  Since we weren’t familiar with C#, we looked for alternatives with JavaScript and other methods to incorporate our API code which was in Python.
Another challenge we ran into was learning how to utilize Microsoft Azure. Although the workshop gave a really good overview of the services it offered and its practicality, we still had some difficulties deploying our own projects or learning the API documentation for the purpose of our project.

Accomplishments that we're proud of

We are proud of our combined team effort in completing our assigned roles, each person working towards their strengths, and our efficiency working in a fast-paced, stressful environment to produce a not only functional, but impactful product.

What we learned

Not only did we learn new skills like working with APIs, we also gained greater proficiency in current skills like python and html coding. We also learned how different computer applications and coding languages interact to produce interactive apps.

What's next for Talk2Me

In the future, we wish to incorporate a better API for quicker communication between user’s speech input and transcription. There was a speech to text API availability in Azure library however it was beyond our budget and we weren’t able to deploy the service. If we had more resources, that would be the API we would’ve used.
We also wish to expand beyond English language to target a broader range of language learners in the future.
Another possible expansion is Incorporate machine learning and artificial intelligence to allow users to practice their English as if they’re speaking to a native.