Horus

Inspiration

As a team, we're all close with our grandparents. As they get older, we want to be able to support them in any way that we can, even when we can't be there. Our ideation focus started out on elderly care homes--we wanted to devise a system to help keep an eye out for elderly patients, especially those who could be prone to falling or wandering off due to a condition like Alzheimer's.

But we realized that interactive and accessible security isn't just limited to elderly care homes. From consumer usage with parents, to enterprise-level usage at companies that need real-time security monitoring, interactive security is largely lacking.

What it does

Our pipeline takes in live stream of videos, and leveraging zero-shot action recognition, we're able to provide real time feedback on chunks of videos that are of interest. Users are able to input with natural language, what they would consider "flaggable" behavior. We match chunks with flaggable behavior, and then, if an incident is flagged, real-time deployment of Twilio API calls the user who deployed the software. The user is then notified of the incident, and able to talk to Twilio about 1. Context of the incident 2. Possible next steps to take 3. Extractive video

How we built it

Computer Vision.

Azure integration, to provide holistic view on people actions during frame, with natural language. Multi-threading allows us to achieve this in real-time without much latency. Chunking video into actions from the stream to be saved into our cache. Cache detection runs through GPT, and submits a POST request to our flask server when an action is deemed to be flaggable.

Twilio retrieves.

Twilio

Separate tunneled local server on ngrok so Twilio can access. Then, server receives live requests that uses TwixML to give appropriate responses to users. Utilizing whisper, users are able to have real-time conversation with Twilio, which has access to custom defined toolkits.

Challenges we ran into

A lot of networking challenges. Some API keys disappeared here and there (we reached our limit), and spent like 2 hours debugging for no reason.

Accomplishments that we're proud of

As beginners at our first hackathon, we really got to get hands on with building a project, and become a super cohesive team!

This was a lot of our first times fully building out a deployable application, and it was really great just learning how to take our skills in class to apply to the real world. It's super inspiring to know that what we make can change the world.

What we learned

So so much. From multi-threading to lightsaber-duels, the knowledge was everywhere.

What's next for Horus

Huge emphasis on stopping implicit bias, especially in CV algorithms. We planning to focus on the ethical implications of software like Horus, stressing that implicit biases don't make their way into the input, detection, or communication pipelines.