Aegis Security Agent

Multimodal AI Co-pilot with Real-time Security Workflows using Multi-Agent Reasoning and Natural Language Control

Comment 4

Inspiration

Security is a growing concern in today’s world, especially in public spaces, campuses, and workplaces. Our team noticed that while there are many surveillance systems, most require constant human monitoring and lack intelligent, actionable insights. We wanted to create a system that not only streams video but also helps security teams proactively detect threats, analyze situations, and manage incidents—all through a natural, conversational interface.

Each of us has experienced situations where we wished for smarter, more responsive security—whether it was on a university campus, at a public event, or in a residential building. We asked ourselves: what if security teams could have a digital co-pilot that automates all complex workflows - watches, analyzes, and responds in real time ?

What it does

We built AEGIS Security Co-Pilot, a comprehensive security dashboard that combines live video feeds, advanced object detection, scene understanding, and a conversational assistant. The system displays a 2x2 grid of real-time camera streams, provides instant security analysis for each feed, and allows operators to interact with an intelligent agent via a chatbox. The agent can answer questions, run security scans, log incidents, and provide actionable recommendations—all in natural language.

How we built it

We developed AEGIS as a modular web application, with a Python FastAPI backend, a modern HTML/CSS/JavaScript frontend, and deep integration with Google’s Agent Development Kit (ADK) for agent orchestration. Here’s how we approached the build:

System Design & Planning:
We started by mapping out the user journey for a security operator, identifying the need for real-time video, instant analysis, and a seamless chat interface. The architecture was designed to be modular, so new tools and camera sources could be added easily.

Backend Development:
We set up a FastAPI server to handle video streaming, API endpoints, and static file serving. Google ADK was integrated to power the conversational agent, using Google Gemini for natural language understanding. We created endpoints for video streams, real-time frame analysis, camera status, incident logging, and chat relay.

Tooling & Security Modules:
We implemented YOLO for object detection, enabling the system to spot people, vehicles, weapons, and suspicious objects in each frame. A Vision-Language Model (VLM) was added to provide scene understanding and answer natural language questions about what’s happening in each feed. All tools were registered with the agent, allowing it to chain them together for complex queries.

Frontend Dashboard:
The dashboard was built with HTML5, CSS3, and vanilla JavaScript. It features a responsive layout, live camera feeds, real-time analysis, system status indicators, and a chatbox for interacting with the agent. The chatbox connects to the backend using asynchronous fetch calls, enabling real-time communication.

Agent & Tool Orchestration:
The AEGIS Security Agent was built as a Google ADK LlmAgent, with a detailed prompt and access to all security tools. The agent can parse natural language commands, decide which tools to invoke, chain multiple tools for multi-step operations, and return structured, actionable responses.

Database & Session Management:
We used SQLite for lightweight, persistent storage of user sessions and incident logs. The backend manages session creation and retrieval for each chat interaction, ensuring context is preserved across commands.

Testing & Iteration:
We tested the system with both real and demo camera feeds, including YouTube live streams. Quick command buttons and demo scripts were added to showcase key features. The UI and agent responses were refined for clarity, speed, and professional presentation.

Challenges we ran into

One major challenge was integrating real-time video analysis with natural language command processing. Ensuring that the agent could understand complex queries and chain together multiple tools required careful prompt engineering and extensive testing.

Another challenge was managing performance and reliability, especially with multiple camera streams and concurrent user interactions. We optimized the backend for low latency and implemented fallback mechanisms for video streams.

Finally, designing a user interface that is both powerful and intuitive took several iterations. We wanted security operators to feel empowered, not overwhelmed, so we focused on clear visuals, actionable insights, and a conversational workflow.

Accomplishments that we're proud of

We’re proud to have built a system that brings together live video, advanced analysis, and natural language interaction in a single, cohesive dashboard. The seamless integration of Google ADK, YOLO, and VLM allows security teams to get instant answers and recommendations, improving both safety and efficiency.

We’re also proud of the modular architecture, which makes it easy to add new tools, camera sources, or analysis capabilities. The feedback from our beta testers—security professionals and campus staff—has been overwhelmingly positive.

What we learned

Building AEGIS taught us the importance of modular design, prompt engineering, and user-centric workflows. We gained hands-on experience with Google ADK, FastAPI, and real-time video processing. We also learned how to balance automation with user control, ensuring that the system is both powerful and trustworthy.

What's next for AEGIS

AEGIS has the potential to scale with Google in a multitude of industries. Industries include education, warehousing, casinos, and anywhere where there is a large gathering of people. We wish to launch the pilot with warehouses as one of our team members owns a couple warehouses.

We plan to expand AEGIS with integration to access control systems. We’re also exploring mobile and tablet interfaces, as well as support for additional languages and accessibility features.

Our goal is to make AEGIS the go-to platform for smart, responsive security—empowering teams to keep people and places safe, everywhere.

Submission Info: Automation of Complex Processes in North America

Built With

+ 2 more
Share this project:

Updates