TextTone

Inspiration

Often, people struggle to interpret the underlying emotions of text messages. After all, written language is merely a string of words that, when spoken in varying tones, can have profoundly different meanings. Professor Albert Mehrabian, a body language researcher, discovered the 55/38/7 rule, where 55% of all communication is conveyed through body language, 38% through tone, and only 7% through the actual words. With text, at least half of all communication is lost! We've all experienced confusion when trying to understand the intended tone in text messages. For example, while playing GamePigeon, a game within the text message app, with my friend, I won a closely contested game. Afterward, my friend texted me, “I hate you.” I was very confused at that moment. Was he teasing me, or was he genuinely upset? This uncertainty prompted me to immediately call him for clarification. It turned out he was just joking, engaging in friendly banter. Another instance occurred when texting an acquaintance after a school orchestra concert. I complimented one of my fellow musicians, saying, “Good job at the concert; your part sounded perfect!” Their reply was, “Stop it.” Reading that response, I couldn't discern their exact feelings. Were they offended, upset about something else, busy, or simply joking? It turns out they were expressing courtesy towards my comment and were genuinely pleased. After numerous experiences like these, we realized there must be a better way to communicate through text. With 37% of communication relying on the tone of speech, our app aims to facilitate more cooperative and understanding interactions for all parties involved.

What it does

Our app will provide real-time emotion detection using the front-facing camera so that both parties know exactly how the other is feeling while texting online. When each text message is sent, there will be a small facial expression indicating the tone of the message depending on the sender’s facial expression. This will allow the recipient to easily and correctly determine the sender’s meaning.

How we built it

For the frontend, we used MIT App Inventor, a software that allows users to create custom Android apps. The primary reason we used this software is because it allows us to build a realistic, functioning prototype so we can focus on the features we would want to incorporate. It also allows us to use different APIs in the backend to help us achieve our goal. For extensions, we tried using Visual Studio Code, including Javascript and html. These helped with the API integration. MIT App Inventor’s built-in camera function and FaceAnalysis API together, can label the user’s expression as a certain emotion and display a certain emoji on our app for the recipient of the user’s messages to see. For the interface, we tried our best to make it seem as modernistic and simplistic as we could, keeping ease of use in mind.

Challenges we ran into

In the initial stages, our team exhibited strong performance, smoothly navigating the brainstorming session and settling on what seemed to be a feasible project. However, our optimism hit a roadblock when we encountered challenges with the APIs we intended to use. The primary issue surfaced during attempts to input an image from the MIT App Inventor platform to facilitate emotion prediction by the API. Also, joining this with the extensions of MIT App Inventor were particularly tricky. Despite dedicating nearly 5 hours finding a solution, we couldn't bridge the gap. Our focus then shifted to transforming our image into a URL, but this endeavor proved to be an intricate programming task. Faced with time constraints, we reluctantly abandoned the idea as the URL generation process became overly complex after initial attempts. This setback posed a significant hurdle, destroying lots of progress.

Accomplishments that we're proud of

Despite encountering a significant challenge with the API, preventing our app from reaching its full potential, we take pride in the strength of our concept. While a seemingly straightforward idea, TextTone is exceptionally viable, user-friendly, and accessible across various communication platforms. Our belief in the app's potential to benefit a broad audience remains unwavering. Despite the setback, we are determined to refine and overcome the obstacles, aspiring for TextTone to evolve into a tool that brings valuable benefits to numerous users in the future.

What's next for TextTone

The future for TextTone is extremely bright due to the vast digital communication apps that are used every minute by people across the globe. In the future, TextTone envisions expanding its emoji-based emotion detection feature to all digital communication platforms. This includes seamless integration with popular messaging apps such as iMessage or Discord, collaboration with social media platforms such as Snapchat or Instagram to enrich emotional expression in posts and comments, and cross-platform functionality spanning emails, collaborative tools, and more. Users can anticipate customizable emotion sets for personalized expression, enhanced privacy controls, and continuous innovation guided by user feedback. The goal is to make TextTone a universal tool, transforming digital communication by providing users with a consistent, intuitive, and emotionally resonant experience across a diverse range of digital platforms.