ConvoCraft

Inspiration:

The inspiration behind "ConvoCraft" was born out of the idea that meaningful communication is the gateway to creating lasting memories and forging cross-cultural connections during travel. We recognized the universal struggle many travelers face when navigating foreign lands without knowing the local language. The concept was simple yet powerful - create a tool that bridges these linguistic gaps, empowering individuals to communicate confidently while learning on the go and creating unforgettable memories.

The project is the product of the personal struggles and experiences of the members of our team with learning languages - we're all at least bilingual, with some of us speaking 7+ languages! And the one thing that helps most is also the scariest - a language learning conversational partner. So we took the fear out of the equation, added an infinite supply of patience, and made our own!

What it does:

"ConvoCraft" is not only a language communication training product but also a tool for crafting meaningful memories. It offers a safe space for users to practice conversations without judgment. It provides subtle cues and emojis to guide users to those pivotal 'gotcha' moments in conversations, fostering confidence and language learning. ConvoCraft puts you in simple social situations (e.g. you wish to order food in a restaurant), has an AI that only speaks to you in your target language, and gently guides you towards improving your vocabulary, sentence construction and grammar. At the end of a conversation, you can ask ConvoCraft for feedback and it'll tell you what you did well and what areas warrant improvement. ConvoCraft keeps track of your feedback over multiple chats safe in MongoDB Atlas, and can generate an on-demand, comprehensive report of your progress using it. Your ConvCraft partner doesn't just text you back - it talks to you thanks to the superpowers from the Google Cloud Text-to-Speech engine - making it a multi-modal, highly immersive experience.

How we built it:

We are a team of language experts, technologists, and designers who collaborated to create a user-friendly, intuitive device. It relies on advanced algorithms to offer subtle cues and emojis, guiding users in real-time conversations. The design was centered around user feedback and real-life experiences to ensure that learners could practice comfortably.

We minimized code and infrastructure overhead using platforms like glitch.dev and building with the Fastify framework for NodeJS, and built the offering around OpenAI's ChatGPT and Google Cloud Text-to-Speech. The highly tunable and extremely coherent results from GPT-4 combined with the blazing fast speed of GCloud Text-to-Speech not only made this product possible, but made it fast enough to be a real time experience far superior to many language learning experiences that we have tried so far. MongoDB Atlas was used as a persistent data store for tracking feedback from the AI bot that could then be presented as a comprehensive indication of progress.

Challenges we ran into:

We had to overcome connecting various APIs from different services, optimize prompts for content generation, and create a simple yet engaging user interface. Speed was of the essence for a smooth interaction. Additionally, we had to navigate cultural nuances and language variations to make the product effective across diverse regions and cultural styles of communication. Prompt engineering was an interesting challenge in this context since the languages used by the learner and the bot are different, and the bot loses it's utility if it resorts to using English. Eventually, we ended up with something that manages to slowly cement new elements of a foreign language while using almost no English at all.

However, the most special challenge wasn't even a technical one - it was one of introspection as the 4 of us came to an agreement about what really accelerates something as ridiculously difficult as learning a foreign language.

Accomplishments that we're proud of:

We are immensely proud of creating "ConvoCraft," a product that empowers travelers to confidently communicate in foreign countries, facilitating meaningful connections, and forging unforgettable memories. "ConvoCraft" has made it possible for individuals to overcome language barriers, enhance their language skills, and preserve their travel memories in a personal journal.

We personally tried this with our Spanish speaking member trying to familiarize himself with a very challenging language that he hadn't ever seen before - Arabic - and within 10 mins he was able to infer simple things (like the word for "food") and respond with things like "yes" and "no") in the context of the conversational setting. This is an incredible feat when picking up a language so different from your native one!

What we learned:

Throughout the project, we learned that effective language learning goes beyond grammar and vocabulary. It's also about context, confidence, and the 'aha' moments in conversations. Our journey taught us that the most valuable aspect of communication is often the non-verbal cues and the encouragement that comes with understanding and being understood, which are all important aspects of building cherished memories. This was a result of deeply meditating on our own experiences, and recognizing the role of a conversation partner.

What's next for "ConvoCraft":

Expanding its language support and enhancing its features based on user feedback. We aim to continue making language learning accessible, enjoyable, and effective for travelers worldwide, ensuring that "ConvoCraft" remains the go-to tool for cross-cultural communication and memory creation. We also wish to increase the immersiveness of the platform by introducing more modalities to supplement the environment and learning context (e.g. images, videos, animations, VR rooms, etc).

Built With

• axios
• css
• fastify
• glitch
• google-cloud
• gpt-4
• html5
• javascript
• mongodb
• node.js
• openai