Inspiration
The University of Ottawa prides itself on being a bilingual university, offering both classes in French and in English. However, while students are able to learn in their preferred language, unformal communication (text messages and voice memos) between students is still limited to a common language. As this is often English, native French speakers do not get to communicate in their preferred language. To make communication more accessible in everyone's preferred language, we aimed to create a messaging app that would automatically translate incoming messages and generate a voice memo in the likeness of the sender, all in the recipients preferred language. Therefore allowing students to listen to text messages in their preferred language in the voice of their friends!
What it does
Duality is messaging web app that allows you to read and listen to text messages in your preferred language, but in the voice of the sender. For example, if the user's preferred language is French, incoming English text messages will be translated to French and a audio recording of the French text will be generated using a voice clone of the sender.
While our original goal was to make communication in ones preferred language more accessible, Duality also has other use cases:
- It can help speakers with speech impairments communicate more effectively by generating speech in their own voice
- Users can listen to text messages in a foreign language that they are trying to learn, with the ability to compare to the original
- Users can listen to what they sounds like speaking a foreign language to improve pronunciation while learning
How we built it
The frontend was built using React and JavaScript, while the backend was built using python and Node.js. The voice cloning was done using a xTTSv2, a text to speech foundation model that is freely available from Coqui-ai's TTS library. The text to speech model was hosted on a docker container.
Challenges we ran into and what we learned
Setting up an AI model on the backend was unfamiliar to us and was a very rewarding experience. We learned about Firebase, Firestore, and Cloud Storage using buckets, which were all unfamiliar concepts before the Hackathon.
Accomplishments that we're proud of
It was fun to listen to our own voices speaking other languages and with different accents!
What's next for Duality
Our original plan of making mobile app had to be pivoted to a web app because of the time constraints of the hackathon. A natural next step for Duality would be to make a mobile app with the same features or polish the UX of the web app.
Footnotes
This project was made by 2 UBC students (and 1 UOttawa student), and brought to you by Tuum Est!
Log in or sign up for Devpost to join the conversation.