Watchdog

Demo of WatchDog dashboard

Inspiration

During natural disasters or emergencies, responders often miss out on critical information. One way this can be solved is repurposing existing security cameras to automatically detect emergency situations, and send critical data about the situation to first responders immediately. This system could save responders precious seconds and even minutes in emergencies that could mean the difference between life and death.

What it does

Our product WatchDog will run real time AI vision on existing security cameras and automatically detect danger levels based on what is happening. It will also locate all the people it sees, and map them out to a 2D grid - effectively revealing the location of all possible victims in an emergency to EMS. Now instead of searching aimlessly or spending time investigating, responders will know exactly where the threat is occurring and where the victims are. All of this information is displayed on a central dashboard for the emergency responders. Furthermore, when high levels of danger are detected the system will send out automatic texts to key personnel.

How we built it

Overshoot AI receives our specific prompts and outputs the detected danger level, a summary of the scene, and grid-mapped locations of any people in the scene. This is done by mapping the 3D image to a 2D grid. This is then packaged and displayed in a Next.js web-app that displays the information in a dashboard style. Twilio is used to automatically send SMS texts when high danger levels are detected. The entire application was made using TraeIDE, where we were able to get AI assistance that allowed us to make progress more rapidly.

Challenges we ran into

1) 'Danger' is difficult to classify. During testing, we had to deal with false positives and false negatives. We dealt with the false positives by chaining - we only messaged responders if we had the same danger signal for 3 clips in a row. We dealt with false negatives with prompt engineering towards identifying dangerous behavior, such as alarmed facial expressions and fights.

2) We had to balance the tradeoff between prompt length with latency. We did this by optimizing prompt length and customizing overshoot clip length and clip delay settings.

3) We planned to incorporate the ability to reply to our system SMS messages to get certain information, but this would involved registering a Twilio number and would likely take too long to complete.

4) We also planned to incorporate multiple cameras into the same dashboard. We got views from different devices to display in the web app, but didn't have time to polish the implementation.

Accomplishments that we're proud of

We are proud of the comprehensiveness of this system, handling everything from an AI summary to a mapping of peoples' locations, and even customized SMS alerts.

What we learned

We learned about how to get the most out of APIs like Overshoot by varying our prompts to be more specific, and hone in on prompts that yielded the best results. We also learned about how to interface web-based systems with personal phones through SMS messaging.

What's next for WatchDog

Next we want to improve the SMS communication by allowing users to interact with the notification system to get tailored information. We also want to expand our dashboard to display multiple views from several cameras, and provide custom information about each view.