Communico

GIF
GIF
GIF
Chat transcription feature leveraging text-to-speech
GIF
Emotion based classification system

💡Inspiration

The inspiration behind this was Affan's first job working as an inclusion and integration instructor for his city. At his job, he helped a kid with low-functioning, non-verbal autism spectrum disorder who used an AAC app on his iPad. After working his job, he realized how cumbersome the whole system was, how hard it was to understand his kid, and how difficult it was to learn for others. Ever since then, he was thinking of ways to improve the AAC space, and DeltaHacks became the perfect opportunity for this to become reality.

🗣️What it does

Communico is an all in one Augmented and Alternative Communication app that allows nonverbal or differently abled users to express themselves using picture-based tiles, sentence building, drawings, transcriptions, and AI-assisted conversation support. Users can visually construct sentences, hear them spoken aloud, receive real time next word tile suggestions, convert drawings into simple sentences, and transcribe complex speech or text into AAC-friendly tiles. The app also suggests short, context-aware replies to help users participate in conversations quickly and confidently.

⚙️How we built it

Tech Stack

Built using React Native with Expo (Expo Go during development)
Used TypeScript for cleaner and safer code
Structured the app with screens, components, hooks, and services

Emotion Detection

Implemented face-api.js (TensorFlow.js) for real-time emotion detection
Ran emotion detection inside a WebView since TensorFlow requires browser features
Processed webcam frames every ~250ms
Detected 7 core emotions
Cached ML models locally to avoid reloading each time
Sent emotion data to the app using WebView → React Native messaging

Camera Handling

Used expo-camera for webcam access
Handled permissions and start/stop logic to prevent crashes

AAC Symbols (Picture Tiles)

Sourced symbols from OpenSymbols
Stored and cached symbols locally
Used lazy loading so tiles only load when needed
Optimized image sizes to improve performance

AI & Responses

Used OpenRouter API to generate short, emotion-aware responses via Gemini
Passed detected emotion as context for better replies
Streamed responses for faster feedback

Text-to-Speech

Integrated ElevenLabs for natural voice output
Played audio with expo-av
Cached common phrases to reduce repeated API calls

Performance & Data

Cached images, audio, and models to reduce load times
Used AsyncStorage for local settings and message history
Kept state management simple with React hooks

⚠️Challenges we ran into

Balancing feature richness with simplicity and accessibility so the app didn’t feel overwhelming for AAC users
Loading a large number of open-source AAC symbols and correctly saving and caching them locally without slowing down the app
Managing real-time emotion detection while keeping performance smooth and responsive
Handling camera permissions, device compatibility, and platform differences between Android and iOS
Coordinating real-time data flow between the UI, AI suggestions, and user input without lag
Ensuring AI-generated suggestions were helpful but not intrusive, keeping the user fully in control
Debugging and integrating multiple APIs simultaneously under hackathon time constraints

🏆Accomplishments that we're proud of

Building a fully functional, multi-modal AAC app in a span of 24 hours, supporting picture tiles, sentence building, drawing input, transcription, and AI-assisted conversation
Successfully integrating multiple APIs into one cohesive system to work in real time
Implemented live webcam-based emotion detection and connected it meaningfully to the communication experience
Designed and implemented lazy loading and caching for image tiles and audio so the app stays fast and usable without repeatedly reloading assets
Created a project focused on real social impact, inspired by firsthand experience with AAC users and inclusion work
Consumed more than 1000 mg of caffeine combined within the span of 24 hours

📚What we learned

Good AAC design starts with understanding real user needs, not blindly adding features
Built a multi-feature AAC app in React Native with Android & iOS support via Expo Go
Learned how to manage real-time UI updates while keeping the interface accessible and easy to use
Gained experience integrating multiple APIs under one infrastructure
Used OpenRouter with the Gemini API for:
- Next-word suggestions
- Sentence simplification
- Context-aware replies
Integrated face-api.js for live input and emotion processing
Used ElevenLabs for clear text-to-speech and speech-to-text
Used OpenSymbols to source and organize AAC picture tiles
Implemented lazy loading and caching so picture tiles and generated audio are stored locally and don’t reload each time
Learned that AI works best in accessibility tools when it supports the user rather than taking control

🚀What's next for Communico

First of all, we want to deploy our app in a free open-source manner so this can be used by anyone across the world. We want to expand Communico beyond one way communication into shared experiences. One major goal is building a direct messaging feature over multiple devices that allows multiple Communico users to chat with each other using picture tiles, sentence boards, and suggested replies. We also plan to continue development with research backed AAC practices where we'd work alongside speech language pathologists, educators, and caregivers to refine vocabulary sets, layouts, and AI behavior. This includes improving personalization so the app adapts to each user's communication style, preferred words, and daily routines. Additional future improvements include multilingual support, offline-first functionality, better emotion and context awareness, and caregiver/therapist configuration tools. Long term, we envision Communico as a flexible communication platform that grows with its users and supports them across school, work, and social life.