Allegra AI

Inspiration

Growing up, music was a fundamental part of my life. Whether it was singing with my dad after he got off of work each day or blasting Billy Joel after I first got my driver's license; music has been fundamental to my development into a multilingual person.

Every time I become addicted to a song in a target language I am learning, my first goal is to translate the song, understand it word by word, and decode the grammatical structure of each line. But doing this manually for all 1109 songs in my playlist became too tiring too quickly, which sparked my inspiration to create Allegra AI. Moreover, I found that leveraging both traditional book study methods with immersive cultural immersion optimizes language acquisition, which Allegra AI aims to incorporate into its model.

## What it does The user creates an account with Allegra AI and initiates their learning by uploading one of their favorite songs in Spanish. Allegra then converts the speech into lyrics and prompts GPT4 to create questions for the user. The user then answers fill in the blank questions about the song and the app computes the user’s accuracy. From there, the user has the option of asking Allegra for help on any general grammatical question, where it then queries the vector DB using a similarity search method until it finds the concept that the user desires to study.

How I built it

To create this product, I used a Pinecone vector database, text-embedding-3-small, GPT4, Whisper 2, SKLearn k-mean nearest neighbor Machine Learning, Google page rank algorithm, Flask, SQLite, HTML5, CSS, vanilla.js. I first created a login system using SQLite to give each user access to their own unique account. The site then requests the song as an mp3 file from the user and employs the OpenAI Whisper model to convert it to text—GPT4 is then deployed to prepare questions from the lyrics surrounding grammatical concepts and vocabulary. Once the user answers the final questions, the vector database queries what topics the user wants to study (can be any question). I then ran a Sklearn KMeans clustering as well as the Google page rank algorithm (updated once a day) to determine how well the user is progressing on the leaderboard against other users.

Challenges I ran into

Some challenges I ran into was time. Originally, I had planned this application to be more personalized to the user, but because I was cut short on time, I had to keep features generalized instead of a customized learning platform that relied on a user’s strengths and weaknesses. Another challenge I ran was vectorizing the song itself, because I initially believed that the user could learn new grammatical concepts from lyrics. I quickly realized that this wouldn’t be enough for the user, which prompted me to use a LibreTexts Spanish textbook to supplement the user’s learning.

Accomplishments that I am proud of

Some accomplishments that I am proud of is the leveraging of technologies that I knew a little of but mastered after creating Allegra AI. I am proud to have created a powerful application that supplements a user’s immersive cultural learning experience with Spanish. Most importantly, I’m glad to have made it out of this project alive and created an optimal product that will propel a user into fluency with Spanish.

What I learned

A professor told me that back in the day, he was coding in SQLite server. Curiosity took over, but I never really worked with the library. This hackathon really allowed me to meddle with SQLite. Something that I never really thought about was how Google indexes their web pages. After looking for methods that would rank a user by correctness on their exercises, I learned about the Google page rank algorithm and applied it to this project which came out successful.

What's next for Allegra AI

My first course of action is to expand this platform to more languages in addition to Spanish to attract more users. This way, a diverse set of learners can join this platform and interact with other users in their language learning journey. I also hope to gamify Allegra AI more, by incorporating mini Spanish competitions where users compete with each other on their knowledge of their favorite songs and are ranked.

Built With

• css3
• gpt4
• html
• javascript
• ml
• openaiapi
• pinecone
• python
• sqlite
• translateapi