StreamLens

• 

  Architecture

• 

  Q&A for the video (RAG)

• 

  Highlight reel generation

• 

  Live scene understanding with summary

• 

  Highlight Reel is generated which is 1/4 of the original video

StreamLens Pitch

Inspiration

• Every year, a staggering amount of content is streamed across various platforms:
• NFL, soccer, cricket, and more, contribute to thousands of hours of sports content. Webinars and events add to the growing pool of educational and entertainment material.

Currently, the creation of video content and metadata is a manual, time-consuming process. We saw an opportunity to streamline this, making content more accessible and engaging.

What it does

StreamLens revolutionizes how we interact with streaming and static video content. By leveraging advanced AI, it can understand, summarize, and generate highlights of video content in real-time. This includes:

• Live Context Understanding: Dynamically comprehends the content being streamed, providing immediate insights.
• Key Moments Summarization : Identifies and summarizes the most significant moments in any game or event.
• Q&A Feature: Allows users to ask questions and receive pinpointed answers, directing them to relevant moments in the content.
• Automated Video Highlights Generation: Creates highlight reels automatically, saving countless hours of manual editing.

How we built it

We utilized a combination of cutting-edge AI models and user-friendly front-end technologies:

• Models : Integrated ASR (BentoML Cloud) for audio recognition, NeVA (Cloud API) for video analysis, and Mixtral (Cloud API) for a comprehensive multimedia content understanding.
• Frontend: Developed using Streamlit, ensuring an intuitive and responsive user experience.

Challenges we ran into

• API Deployment: Mastering the deployment of BentoML cloud API for seamless model integration.
• Unified Content Analysis: Merging audio transcription and vision/scene understanding into a cohesive analysis tool.
• Multimedia Processing: Addressing technical challenges related to streaming, callbacks, and efficient processing of audio and video files.

Accomplishments that we're proud of

• Architecture: Designed an innovative application architecture that seamlessly integrates multiple AI technologies.
• Potential: Demonstrated the transformative potential of applying AI to streaming content, enhancing user engagement and automating content curation.

What we learned

• AI Application: The application of AI to streaming data presents unique challenges but offers significant rewards.
• On-device Inference: The advancements in on-device inference are opening new possibilities for real-time content analysis.

What's next for Multimodal AI

• Enhancing the Q&A feature using semantic search to guide users directly to key moments, improving the interactive experience.
• Make the automated highlight/reel generation part of the UI, which are separate apps now.
• Refining AI models for greater accuracy and a broader range of content analysis capabilities.

Built With

• astra
• bentoml
• llamaindex
• mlx
• neva
• nvidia
• whisper