Inspiration
- The "Active Mirror" (The Creative Flow) Think of the "Rubber Ducking" method programmers use—talking to an object to solve a problem. EchoMate isn't a duck; it’s a mirror that talks back.
The Scenario: You’re pacing your kitchen, describing a complex app architecture or a movie plot.
The Feature: Live Mapping. As you speak, Gemini Live identifies key entities and relationships, instantly generating a visual diagram or a structured outline on your connected screen (laptop/tablet) in real-time.
The "Wow" Factor: You stop talking and say, "Wait, go back to that third point," and it instantly recalls the exact context of your verbal thought from three minutes ago.
- The "Eyes on the Ground" (The Visual Partner) Leverage the Gemini Live Multimodal capabilities (Camera Sharing).
The Scenario: You are fixing a leaky faucet or assembling a complex PC build. Your hands are covered in grease or holding tiny screws.
The Feature: Over-the-Shoulder Expert. You point your phone at the mess. You don't have to ask "What is this part?" You just talk through your frustration. EchoMate sees the part, hears your tone, and says, "I see the issue—the O-ring is seated backward. Try flipping it while I hold the manual open for you."
The "Wow" Factor: It feels like a master craftsman standing right next to you, not an app you have to query.
- The "Infinite Memory" (The Context Bridge) Use Gemini 1.5’s massive context window to make EchoMate a "Long-Term Mate."
The Scenario: You’re in a live session and say, "Remember that idea I had last Tuesday while I was driving? How does this new thought fit into that?"
The Feature: Temporal Synthesis. EchoMate pulls from your history of "Live" sessions to find contradictions or synergies in your brainstorming over time.
The "Wow" Factor: It proves that the "Live" experience isn't just a one-off phone call; it's a persistent, evolving relationship with your ideas.
- The "Vibe Sync" (Emotional Intelligence) Hackathon judges love "Human-Centric AI."
The Scenario: You’re nervous about an upcoming pitch and you’re practicing out loud.
The Feature: Adaptive Rhetoric. EchoMate analyzes your speech patterns, pace, and sentiment. If you’re stuttering, it slows down its own voice to calm you. If you’re high-energy, it matches your hype to keep the momentum going.
The "Wow" Factor: It doesn't just provide information; it provides encouragement and critique based on your current state of mind.
What it does
- Real-Time "Thought-to-Asset" Generation While you are talking to Gemini Live, EchoMate is working in the background to build things.
The Action: You’re walking around your office describing a new app.
The Result: EchoMate is simultaneously generating a Mermaid.js flow chart, a Trello board, or a Markdown document that populates on your screen as you speak.
The Hack: It uses Gemini's function calling to trigger external APIs (Notion, GitHub, Slack) based on the intent of your live speech.
- Visual Co-Pilot (See what I see) It uses the camera feed to bridge the gap between "saying" and "doing."
The Action: You point your phone at a whiteboard with messy scribbles.
The Result: EchoMate says, "I see your logic for the database schema. Based on what you just said about scalability, should we move the 'User' table to a microservice?" * The Value: It treats your physical environment as part of the conversation context.
- Contextual "Echo" (The Memory Bank) It solves the "What did I just say?" problem.
The Action: You’ve been brainstorming for 20 minutes and get interrupted by a phone call.
The Result: When you return, you ask, "Where was I?" EchoMate doesn't just give a transcript; it provides a Synthesis: "You were debating between a subscription model and a one-time fee, and you leaned toward the subscription because of recurring revenue."
- Interactive "Vibe" Adjustment It acts as a high-EQ coach.
The Action: You’re practicing a high-stakes presentation.
The Result: It tracks your "Live" audio metrics. It interrupts (politely!) to say, "You’re speaking 20% faster than usual.
How we built it
The Core Engine: Gemini 1.5 Flash & Pro We utilized Gemini 1.5 Flash for the primary "Live" interaction due to its sub-second latency, ensuring the conversation feels natural. We toggled to 1.5 Pro for "Deep Analysis" moments—like when the user asks to synthesize an hour-long brainstorming session into a technical spec.
Real-Time Communication: WebRTC & Multimodal Live API To achieve the "Live" experience, we used the Gemini Multimodal Live API.
Audio: Bi-directional streaming using low-latency WebSockets.
Vision: We pushed video frames from the user's camera to the model's context window, allowing EchoMate to "see" and "hear" simultaneously.
- The "Action" Layer: Function Calling This is how EchoMate goes from talking to doing. We defined a set of Tools (Client-side functions) that Gemini can trigger:
: Converts spoken summaries into structured Markdown.
: Extracts action items and pushes them to a project management API (like Trello or Notion).
: Takes structural descriptions and outputs Mermaid.js code for instant visualization.
- Context Management: The "Infinite" Memory We leveraged the 2-million-token context window. Instead of clearing the chat history, we maintained a "Living Context."
The Hack: We used Context Caching for frequently referenced project files or previous session transcripts to reduce cost and speed up response times for long-term projects.
Challenges we ran into
- The "Interrupt" Logic (Voice Activity Detection) The Challenge: In a normal chat, it’s one person at a time. In a Live environment, humans interrupt, mumble, or have background noise. We didn't want EchoMate to stop talking every time a dog barked, but we wanted it to stop immediately when the user had a follow-up question.
The Solution: We fine-tuned the Sensitivity Threshold of the Voice Activity Detection (VAD). We implemented a "Graceful Pause" where the model holds its thought for 500ms before deciding if the user truly took over the floor.
- Latency vs. Intelligence The Challenge: Gemini 1.5 Pro is brilliant but can have higher latency for complex reasoning. Gemini 1.5 Flash is lightning-fast but occasionally missed the subtle "subtext" of a brainstorming session.
The Solution: We built a Hybrid Routing System. Flash handles the immediate "Live" verbal feedback (the "Echo"), while a background worker sends the full conversation transcript to Pro every 60 seconds to generate the high-level "Documentation" and "Mind Maps."
- Managing the "Visual Noise" The Challenge: When the camera is on, the model is flooded with data. If the user moves their phone quickly, the model might get "distracted" by a poster on the wall instead of focusing on the notebook on the desk.
The Solution: We implemented Frame Filtering. Instead of a raw video dump, we used a "Trigger-Based Snapshot" system. EchoMate only "analyzes" a high-res frame when it detects a verbal cue like "Look at this" or "What do you see here?" This kept the context window clean and the responses focused.
- Async Function Collision The Challenge: Since EchoMate is doing things (like updating a Trello board) while talking, we ran into "Race Conditions." The AI would try to summarize a thought before the function call to create the note had actually finished.
The Solution: We created an Event-Driven State Machine. This ensured that the UI and the AI's verbal response stayed "In Sync." The AI is instructed to acknowledge the action (e.g., "I'm adding that to your list now...") only after the API returns a 200 OK status.
Accomplishments that we're proud of
- Zero-Latency "Flow" State We are incredibly proud of the conversational fluidness we achieved. By optimizing our WebRTC implementation and fine-tuning the Voice Activity Detection (VAD), we eliminated the awkward "walkie-talkie" pauses typical of AI voice assistants.
The Result: EchoMate feels like a person on the other end of a phone call, capable of being interrupted and picking up right where it left off without losing the thread of the conversation.
- The "Visual Context" Breakthrough We successfully bridged the gap between sight and sound. We developed a custom "Context Trigger" that allows the AI to reference physical objects in real-time.
The Result: During our testing, we were able to point the camera at a complex, handwritten whiteboard diagram and have EchoMate not only explain the logic but simultaneously generate a clean, digital version in a side-panel.
- Real-Time "Actionable" Intelligence Most AI assistants just talk; EchoMate works. We are proud of our seamless integration of Function Calling within a live stream.
The Result: We demonstrated that the AI could extract a verbal commitment—like "I'll send that email by 5 PM"—and automatically populate a calendar invite and a draft email in the background, all while the user was still speaking.
- Handling the "Long-Tail" of Thought Using Gemini 1.5’s 2-million-token context window, we solved the "Goldfish Memory" problem.
The Result: EchoMate can recall a tiny detail mentioned 45 minutes ago in a rambling brainstorm and connect it to a new idea perfectly. Seeing the AI say, "That actually solves the problem you mentioned at the start of our walk," was a massive "Aha!" moment for our team.
What we learned
- Voice is the "High-Bandwidth" Interface We learned that speaking is fundamentally different from typing. When users talk to Gemini Live, they share more than just data; they share intent, tone, and uncertainty.
The Lesson: We realized that EchoMate shouldn't just respond to words, but to the gaps in speech—the "ums," the "uhs," and the long pauses where a user is actually asking for a nudge or a suggestion.
- The Power of "Low-Friction" Multimodality At the start, we thought vision and voice were separate features. We learned they are a single sensory experience.
The Lesson: When a user says "This thing here," and the AI instantly knows what "this" is because of the camera feed, the "uncanny valley" disappears. We learned that for an AI to feel like a "Mate," it must share the user’s physical context, not just their digital one.
- Latency is the Ultimate "Vibe" Killer We learned that even a 2-second delay turns a "conversation" into an "interrogation."
The Lesson: We spent 40% of our time optimizing for speed. We learned that a slightly "less smart" response that is instant is often more valuable in a live flow than a "perfect" response that takes 5 seconds to generate. This led us to our hybrid Flash/Pro architecture.
- AI as a "Proactive" vs. "Reactive" Tool Most of us are used to AI that waits for a prompt. Through building EchoMate, we learned the value of proactive listening.
The Lesson: We discovered that the most "magical" moments happened when the AI interrupted to say, "I've already started a draft of that for you," before the user even asked. We learned that the future of AI is moving from a "Search Engine" to a "Co-Processor."
What's next for EchoMate
- Ecosystem Integration (The "Action" Expansion) Currently, EchoMate can talk and create basic notes. The next step is turning it into a Universal Controller.
The Goal: Integration with tools like GitHub, Jira, and Slack.
The Vision: "EchoMate, I’m stuck on this function." The AI sees your screen, finds the bug, and asks, "Should I open a PR with the fix?" You say "Yes," and it's done—all through the Live interface.
- Multi-Device "Spatial" Awareness Using Gemini’s multimodal capabilities to move beyond a single phone screen.
The Goal: An "Omni-Channel" session where EchoMate follows you from your phone (while you pace and talk) to your desktop (where it has already opened the relevant files) to your smart glasses (providing a HUD of your notes).
The Vision: A seamless handoff where the "Live" conversation never has to restart just because you changed rooms.
- Personal Knowledge Graph (Long-Term Memory) Moving from a 2-million-token "session" to a persistent digital twin.
The Goal: Building a local, encrypted database of all your "Echoes."
The Vision: EchoMate becomes your personal historian. "Hey, what was that weird idea I had about decentralized coffee shops six months ago?" It retrieves the exact moment, the tone of your voice, and the sketch you made on a napkin.
- Collaborative "Multi-Human" Mode Gemini Live is currently a 1-on-1 experience. We want to expand this to Team Echo.
The Goal: A Live Agent that sits in a physical room with three people.
The Vision: It acts as the ultimate mediator—tracking who said what, resolving contradictions in real-time, and ensuring that by the end of the meeting, the "Echo" (the minutes and action items) is 100% accurate and agreed upon by everyone.
Log in or sign up for Devpost to join the conversation.