Echo | Devpost

Home_Page
Figma_Layout

Inspiration

The inspiration behind our hackathon project comes from the challenges faced by young children with Autism, Dyslexia, ADHD, and other neurodivergent conditions as they navigate a world that is often unaccommodating to their needs. Many of these children also experience Alexithymia—a difficulty in identifying and understanding emotions—which makes interpreting facial expressions and social cues even harder.

Alexythimia

Traditional education systems often fall short in providing the tools and structure these children need to thrive. I’ve personally seen kids singled out for not “reading the room” or understanding social dynamics—not because they lacked potential, but because they weren’t taught in ways that resonate with them. Children learn best through experiences that are fun, engaging, and interactive.

This is why we created Echo. It’s a playful, gamified tool designed to help kids differentiate emotions and expressions—both on their own faces and those of others—in real time. Echo makes this process accessible, enjoyable, and impactful, giving kids the support they need to grow with confidence in a way that truly works for them.

What it does

Echo is a technically sophisticated app designed to assist individuals, particularly children with Autism or other neurodivergent conditions who struggle with Alexithymia, in learning emotional recognition and expression. The app integrates cutting-edge technologies such as computer vision, 3D modeling, and machine learning to create an immersive and interactive learning environment. At its core, Echo combines a polished UI inspired by HackUTD's "Ripple Effect" theme, with a 3D HackUTD mascot and a secondary animated mascot that serves as an interactive guide. This guide offers real-time feedback to users as they engage with the platform. The app features two primary functionalities: a webcam-based emotion training tool, where users mirror the mascot’s facial expressions using facial landmark detection and emotion recognition, and an emotion recognition quiz that challenges users to identify emotions from randomized facial expression images. These images are dynamically pulled from a decentralized file system managed through the Pinata API, ensuring seamless and secure storage and retrieval. The quiz is enhanced with advanced sentiment analysis, providing real-time feedback generated by Samba Nova’s API, which utilizes fine-tuned machine learning models for scientifically backed evaluations.

How we built it

The backend of Echo is developed using Flask, which handles complex routing between the frontend and backend, processes webcam input, and ensures seamless functionality by managing real-time API calls. The frontend is built using React, Next.js, and JavaScript, integrating advanced WebGL rendering to manipulate 3D mascots and create a highly interactive user experience. The mascots themselves are designed in Blender, utilizing intricate rigging and animation techniques to create smooth, dynamic interactions. Additionally, the emotion recognition quiz employs a probabilistic randomization algorithm, ensuring diverse image presentation that keeps users engaged with new and varied emotional expressions. The Pinata API is used for decentralized file storage, while Samba Nova’s API generates sentiment analysis and feedback through machine learning models. With this combination of advanced systems, Echo goes beyond simple coding and merges multiple technologies into a unified experience, showcasing innovation in both design and functionality.

Pinata!

Samba Nova!

Challenges we ran into

Throughout the development of Echo, we encountered several challenges. One of the main hurdles was figuring out how to display images uploaded to Pinata and learning how to effectively use its API. Integrating the webcam into the website and ensuring it could identify emotions at a faster rate also proved difficult. While the system works, we are still working on improving its performance, especially when dealing with multiple people in the frame, as the current speed is slower than desired. Additionally, implementing and using Three.js for the first time was a significant challenge, as it required a steep learning curve to create the desired interactive 3D elements. Our team also faced the pressure of working with newer hackers—two of our members were relatively new to hacking, so they had to quickly get up to speed on GitHub, APIs, and full-stack development while juggling these tasks under time constraints. Another issue was optimizing the facial recognition for low latency, and navigating the documentation for Samba Nova and Pinata APIs added to the complexity. Despite these obstacles, we made steady progress, learning and adapting as we developed the app.

Accomplishments that we're proud of

Some accomplishments we're particularly proud of include creating a visually appealing and highly functional landing page that aligns with the HackUTD Ripple theme. Not only does it look amazing, but it’s also user-friendly, featuring a cute design with a dynamic 3D element that users can interact with by moving their mouse. We're also proud of successfully implementing the webcam functionality and the emotion recognition quiz page, which were key features of the app. Additionally, integrating the APIs into our code was a significant achievement, allowing us to pull images from Pinata and use Samba Nova for sentiment analysis effectively. Overall, we’re proud of how many different aspects of the project we were able to complete and how well they work together. We can truly envision users benefiting from the app, both enjoying the engaging interface and gaining value from the emotional learning tools we've built.

What we learned

Throughout the development of Echo, we learned a great deal about various technologies and best practices. We gained valuable experience with Three.js, Flask, and how to integrate the frontend and backend more seamlessly. Working with a team also taught us how to efficiently collaborate, manage version control, and handle pulling and pushing code effectively. One of the biggest challenges we faced was understanding and resolving issues related to CORS (Cross-Origin Resource Sharing), as we were working with different programming languages on separate local hosts. We ran into problems when the web application attempted to access resources from different domains without proper server permissions, but we learned how to troubleshoot and resolve these issues.

We also deepened our understanding of how to use APIs like Samba Nova and Pinata, learning how to integrate them into our project effectively. Samba Nova, for instance, required us to explore its documentation and fine-tune machine learning models for sentiment analysis, while Pinata helped us handle decentralized file storage. Overall, these experiences expanded our technical knowledge and helped us become more adept at problem-solving under pressure.

What's next for Echo:

Next, we plan to continue improving Echo by implementing several new features to enhance the user experience. One of our key goals is to expand the range of emotions recognized by the app, allowing for a more comprehensive learning experience. We also aim to optimize the performance for faster output, particularly for emotion recognition, to improve user confidence in the app's responsiveness. In addition, we’re looking to add more engaging quiz features, such as flashcards and matching games, to make learning more interactive and fun.

We also plan to expand the app’s capabilities to include voice recognition for tone cues, which will help users who struggle with understanding vocal emotions. To further enhance user engagement, we’re working on adding more dynamic mascot animations, which will enrich the interactive learning experience. Additionally, we want to revamp the reward system to include tokenized rewards, levels, a progress bar, daily streaks, and badges to motivate users and track their progress. These updates will make the app even more tailored to individual needs, ensuring that it continues to be an effective and enjoyable tool for emotional learning.