Callmly

Inspiration

We were inspired by a simple, universal truth: Job hunting is stressful. Interviews are hard, and they often fly by in a blur. We noticed that while many current AI products provide cheats or shortcuts to such inteviews, they can actually harm a user's skills in the long run since they are not actually talking themselves. Our goal was to create a tool that fosters innovative and responsible AI use to help the user grow, rather than just giving them the answers.

What it does

Callmly is an intelligent interview coach designed to fix real-time communication issues that are hard to self-correct. It uses facial recognition and live analysis to monitor your performance as you speak. Key features include:

Pace Tracking: It monitors your speech speed to ensure you aren't rushing.
Vocabulary Analysis: It tracks vocabulary diversity, sentimentality, certainty, and filler word count to help you sound professional and clear.
Health Monitoring: By tracking your heart rate, it helps you practice handling stress in high-pressure environments.
Structure Assistance: It helps you practice response formats like the STAR method.

How we built it

We built Callmly using React and TypeScript for front-end, Node.js to handle back-end, and NLP, Gemini API, and Presage to power the speech analysis and insights.

Challenges we ran into

One challenge was implementing the pace tracking software for the user's speech. We had a problem that the software would only track it when the user stops speaking, and will not update until the user finishes their next sentence. To combat this, we implemented a hard refresh setting every few seconds so that the speech recognition would instantly stop and process the current words per minute.

Accomplishments that we're proud of

We are proud of making the facial recognition software to detect the user's face which worked without any problems. The software initially takes a screenshot of the user's face, then during their live screenshare with a recruiter, it would detect them and monitor only their actions.

What we learned

Working with the Gemini API taught us how to analyze "soft skills" like vocabulary diversity, sentimentality, and certainty, transforming raw data from the user's transcript as well as other metrics recorded into feedback that helps the user grow.

What's next for Callmly

Next, we aim to not only help job hunters improve in their interviews, but also recruiters to see how well their client is doing during an interview, giving them hidden statistics they may not be paying attention to during the interview to better understand their clients and if they are a good hire.