Dial AI

Dashbaord for dispatch
911 call

Inspiration

Alan Liang from Oakland faced a terrifying reality during an emergency. “I called 911, and the experience was horrible. I was placed on hold for 10 to 15 minutes. It took nearly 20 minutes to speak to a dispatcher.” The average wait time for 911 calls in Oakland is 54 seconds, but in emergencies, even a few seconds can mean the difference between life and death. No one should ever have to wait to be connected when every second could save a life.

https://www.nbcbayarea.com/news/local/oakland-911-crisis/3266349/

What it does

The average 911 wait time in cities like Oakland is 54 seconds, but during an emergency, every second counts. Dial AI is designed to support 911 centers when they’re understaffed or overwhelmed with calls. Our system uses a multi-agent approach: one AI agent engages with the caller in real-time instantly with 0 wait time, just like a live 911 operator, while another extracts critical information. Simultaneously, additional agents categorize and prioritize the details for the appropriate departments, ensuring that responders receive the information they need quickly. We provide a comprehensive dashboard to all department responders, where incidents are stack-ranked by severity, complete with the caller's location and all necessary details for immediate action.

In times of high call volume or staffing shortages, traditional 911 lines can become overloaded. Dial AI ensures real-time communication, providing callers with instant responses and swift action. The system scales seamlessly based on call volume, ensuring there are never any delays due to the robust and adaptive design of the technology.

Imagine a scenario like Alan Liang’s—someone in desperate need of help. Instead of being placed on hold for minutes, Dial AI engages immediately, ensuring their issue is addressed without delay.

How we built it

We carefully evaluated different models to ensure the best fit for our use case, understanding that when someone calls 911, they are already in distress. Selecting an empathetic voice model was crucial, and Vapi was the obvious choice for its ability to respond with human-like sensitivity. For real-time conversations, we integrated Vapi with Twilio through webhooks, allowing callers to speak directly into their phones as they would with a live operator.

We built the system using Fetch AI to manage the agents, with AgentVerse hosting them. In this setup, two agents communicate with each other in real time. When a call is placed, the first agent retrieves the chat ID and real-time transcripts using Twilio’s webhook. These transcripts are passed through the agent, which uses OpenAI’s API (GPT-4) to extract key information, organize it, and prioritize the emergency based on urgency. This agent communicates with another agent to pass the extracted information to the dispatcher console.

These agents work together, with one handling the live interactions and event classifications. The data from the first agent is formatted in JSON by the second agent and sent to a dashboard, where dispatchers can view emergencies stack-ranked by severity and location.

By using Fetch AI, AgentVerse, Vapi, and OpenAI, we’ve created a system that ensures quick, real-time responses during high call volumes without delays, making the process efficient, empathetic, and reliable in critical moments.

Challenges we ran into

One of the main challenges we encountered was connecting Vapi with Twilio using webhooks. While obtaining transcripts after the conversation ended was straightforward, extracting real-time transcripts during the call was much more complex. This step was critical for ensuring that information could reach dispatchers as quickly as possible, so overcoming this learning curve was essential for speeding up emergency response.

Another significant challenge was getting agents to communicate effectively within AgentVerse. Since this was entirely new for us, we spent a lot of time learning and refining how agents interact with each other in real-time to ensure a seamless flow of information and decision-making.

Accomplishments that we're proud of

We’re incredibly proud to have built and validated a proof-of-concept AI emergency response agent that offers a truly lifelike experience, making conversational AI over the phone sound remarkably similar to speaking with a real 911 operator.

Beyond that, we’ve developed a complete end-to-end solution that directly addresses a critical real-world issue—reducing high caller wait times while offering vital support to 911 operators. Our multi-agent model performed exceptionally well, and we managed to seamlessly integrate Twilio, Vapi, Streamlit and Fetch AI into a unified architecture.

What excites us most is the real-time conversational AI on phones and how our agents work together to prioritize and process emergencies. This was our first experience working with voice technology, and we’re proud of how natural and effective the interactions feel, as well as how smoothly the agents communicate behind the scenes.

What we learned

We had never worked with voice synthesis or real-time conversational AI until this project, and we’re thrilled to have successfully implemented both. It was also a first for several team members to work with Streamlit and Webhooks, which added new technical challenges and learning opportunities.

Through this project, we’ve gained a newfound respect for the vital work 911 dispatchers perform. It takes exceptional knowledge, skill, and empathy to handle constant emergency calls with such professionalism. While we’re proud of our accomplishments with Dial AI, this experience has deepened our appreciation for the challenging and compassionate work that dispatchers do every day.

What's next for Dial AI

This is just the first step toward creating an ideal world where every emergency call is answered instantly, with zero wait time. While we’re proud of what we’ve achieved, there’s still much to improve. Current models excel in clear-cut, black-and-white scenarios, but they struggle with more nuanced, gray-area situations. With advancements in frontier models, we expect Dial AI to become even better at making contextually aware decisions during calls.

Additionally, we see room for improvement in both latency and empathy. As real-time conversational models evolve, Dial AI will continue to enhance its ability to respond faster and more compassionately.

No seconds should be wasted—every second could save a life. That’s where the real impact of Dial AI lies. While we could have pursued a commercial project, like a customer service AI, our goal was to make a meaningful difference in people’s lives. We wanted to use advancements in technology to improve emergency response and potentially save lives.