RiemannAI

Translate. Summarize. Learn. Using machine learning to bridge language barriers, summarize text, and save time.

Comment

RiemannAI

Translate, Summarize, Learn.

Inspiration

In the United States alone, over 10.4% of all enrolled students are English Language Learning (ELL) students or individuals who face challenges in learning and communicating fluently in the English language [1]. Yet, many of these students – a number close to 5.1 million nationwide – are enrolled in institutions that lack the resources to help them overcome this language barrier.

These ELL students across the country (and arguably the world) struggle to keep up in class due to fast-paced, non-Native language-based classes. Schools may not have access to the necessary human resources and funds needed to accommodate these students, hindering ELL students’ opportunities to learn.

As English as a Second Language (ESL) students ourselves, we have experienced such inconveniences in American schools firsthand, especially in the first few months immediately after moving to the United States.

“I remember when I first moved to the United States in 8th grade. I could not understand any word in science class. I struggled a lot with my grade,” says Cooper Xie, one of our group members.

Thus, when prompted to create something that we “believe will be useful to society”, the thought of tackling this challenge, which our own team members have faced, came to mind.

We were inspired to create RiemannAI, a service that translates and summarizes course content for students worldwide, the very solution many of our team members wished we had access to years ago.

Our Solution

RiemannAI is a platform with a multitude of features designed to translate, summarize, and help students learn.

The key feature of our solution allows students to use RiemannAI to transcribe and translate audio recordings of lectures, discussions, or speeches from most languages into the student’s chosen language. To make the review of the material easier for these students, our solution also incorporates a text summarization capability that returns the main points from this audio recording for the student’s use. The summarized text can be condensed even further into keywords to provide clearer insights to the user.

Yet RiemannAI is more than merely just a single tool. The interactive interface also introduces methods for extractive and abstractive summarization from the text (expanding its functionality to more than just audio input).

RiemannAI's extractive summarization allows students to shorten long, difficult texts to an approachable length by using a custom-created algorithm.

On the other hand, our abstractive summarization uses deep learning based on Google's AllenNLP model to generate a summary based on semantic understanding. In addition to bullet points and a short summary, our abstractive summarization model also allows students to ask a question and get an answer using machine learning. This, in turn, will allow students to approach work completion and studying more efficiently.

Moreover, as students who are passionate about Computer Science, we created a custom API system that allows developers to perform multilingual, extractive, and abstractive summarization.

How we built it

Microsoft Azure Cognitive Services SDK was used to recognize individual sentences and text from the user's microphone. Additionally, live translation was also performed in real-time using Azure WebSockets and Cognitive Services.

Microsoft Azure's Text Analysis SDK was used for extractive text summarization in multiple languages.

Node.js, React.js, Next.js, and ChakraUI were used for frontend and backend web development. ChakraUI was used for styling components and creating a visually appealing site, while the integration of Next.js in the backend and frontend was used to communicate between Azure and the site.

AllenNLP, a model from Google, was used as a base model for abstractive text summarization.

A custom model was programmed in JavaScript to perform extractive text summarization on English text.

LocalStorage was used for storing transcripts and summarizations.

Challenges we ran into

During this project, we discovered that it is difficult to interface with a variety of languages in their written and spoken forms.

Speech Recognition When performing speech recognition, an npm package from React.js was used. However, using built-in speech recognition made it difficult to recognize individual sentences. For example, how would you determine if a user meant to indicate a period or another sentence as a whole?

This task was switched over to Microsoft Azure’s speech recognition instead, which resolved the aforementioned issue and gave us greater uniformity behind the scenes on the website.

Writing Documentation This was all of our team members' first time writing API documentation. However, by putting ourselves into the shoes of client developers, we were able to envision the information they need to fully utilize our API, allowing us to write documentation that is easy to understand and contains examples.

Testing Additionally, testing the accuracy and usability of RiemannAI proved challenging outside of the languages that our team’s members are proficient in. Aside from relying on the performance of the API, we also validated this using outside tools as a benchmark to help us gauge the reliability of the website.

Accomplishments that we're proud of

  • We successfully designed, implemented, and deployed a software architecture that allows for speech recognition, translation, and summarization, with only a few clicks.
  • On top of that, our team came up with an algorithm that makes use of skills in topics such as machine learning and natural language processing. We were able to understand and program concepts like extractive and abstractive summary.

What we learned

  • Cloud Service Providers: By using Microsoft Azure's Cognitive Services and Text Analytics SDKs, we learned useful technical skills used in the real world.
  • Ideation and Creativity: we needed to design a novel solution to a problem that hasn't been solved before. By synthesizing the power of various tools, we grew our ability to think critically and outside the box.
  • Strong communication skills: When designing, building, and deploying our site, we realized that communication between team members was crucial in making sure that everyone was on the same page.

What's next for RiemannAI?

Our team envisions expanding RiemannAI for applications not only in the classroom but as a tool in the workplace, at conferences, and more. In these locations, additional attention to filtering out background noise or non-primary speakers may be needed to fit these settings. As a result, our team hopes to fine-tune the model to contain added functionality on this front.

Additionally, we would like to develop multiple models for both extractive and abstractive summarization. This way, users can choose multiple models for each summarization technique they wish to use.

While the current interface is dependent on having Internet access, we believe it is plausible to introduce a mobile form of this application to increase ease of access. However, all these expansions will require extensive development (and, in some cases, funds) for implementation and deployment.

About the Team

Our team (Ganning Xu, Eric Liu, Qizheng “Cooper” Xie, and Melody Lee) is based in North Carolina. Our members are passionate about linguistics, computer science, and creating tools useful to society.

Works Cited

[1] https://nces.ed.gov/programs/coe/indicator/cgf

Built With

Share this project:

Updates