SpeakEazy

logo

Inspiration

Communication is key -- in all aspects of life. Yet, a fear of public speaking is the most common phobia, affecting more than 75% of people according to surveys. Exacerbated by the lack of social interaction during quarantine, both children and adults struggle with conveying their emotions, especially in front of large crowds.

Often, speakers are so overwhelmed by the nerves of being on-stage that they’re unable to concentrate on effective, emotional delivery, decreasing the impact of their message. What if you had a personalized speech coach and the opportunity for focused practice on emotional delivery, so that not even the worst of stage-fright could phase you? That’s where SpeakEazy comes in, providing personalized feedback about speakers’ emotional delivery.

Yet, even beyond public speaking applications, our tool could help people practice and refine their delivery of important and sensitive personal conversations.

As a group of people who love communication and love helping people learn and improve their skills, the idea for this project emerged naturally.

What it does

SpeakEazy allows users to record themselves speaking and get feedback on the emotions they demonstrated while speaking. After users are done recording, we present users with their results, including the top 5 emotions they displayed.

We also feed their speech transcript and time-stamped data to Chat-GPT to provide intelligent, personalized, qualitative feedback to help them better understand the quantitative expression-data.

By allowing users to practice their speaking with immediate feedback, SpeakEazy allows users to refine their speaking style, identify weak points and hone in on those areas. Additionally, while the LLM can suggest certain emotions that the user should be emphasizing based on the context of their speech, by presenting the complete expression data, we allow users to identify whether or not the emotions they are expressing are intentional or not.

How we built it

We used WebSocket to stream a user's video + audio recording real time to Hume AI's Expression Measurement API.

We also used Chat-GPT's API to create a chat-bot that would enable the user to ask for more detailed feedback on their speaking.

Challenges we ran into

We faced many challenges, especially when trying to integrate Expression Management API and the LLM prompts into our app. While we were able to connect our app to Hume AI, our WebSocket kept closing immediately after connecting.

Accomplishments that we're proud of

We implemented Hume on SpeakEazy which allows for real-time empathic interpretations of facial expressions, tone, and spoken language. This helps public speakers gauge if they are effectively communicating the right emotions to their audience. Additionally, we built a media stream to capture hand gestures and eye contact to pass this information to ChatGPT. This integration allows ChatGPT to generate personalized feedback for the speaker and improve their public speaking abilities.

What we learned

We’ve learned many technical and project-management skills through this project. We also realized the importance of developing with an end-goal in mind, synthesizing different collaboration styles, and the joy of building new connections.

What's next for SpeakEazy

In order to more effectively support users on their journey to overcoming the fear of public speaking, SpeakEazy could incorporate analysis of hand-gestures and non-verbal body language, which are both integral to public speaking and communication more broadly. Additionally, as more complex dynamics emerge when communication is bi-directional, we’re also considering creating a debate-agent that enables two users to record a debate and receive feedback personalized to each user.

We could also see SpeakEazy being integrated into a broader educational ecosystem that leverages game-based learning methods to promote the love of learning among students.