AgentClipz

Inspiration

A lot of people on this team are still in college and relatively young. We watch a lot of streamer highlights in the form of clips on Twitch and YouTube—those short, punchy moments that get shared everywhere. We wanted to help streamers capture those moments automatically instead of scrubbing through hours of VODs or relying on viewers to clip. So we built AgentClipz: an AI pipeline that finds the best moments and gets them ready to share.

What it does

AgentClipz turns a live stream (or a recording) into shareable clips and publishes them. Different parts of the pipeline produce different data: Twitch chat gives us activity and sentiment (as JSON), Modulate gives us transcript + emotions per utterance (JSON), and we may have streamer context or other JSON sources. We also used Yutori in order to scrape trendingg-keywords, those that would be present in trend-worthy clips of the present day. That way, one agent can process and analyze everything together: it sees chat intensity, emotion peaks, and transcript in one place, decides where the “clip-worthy” moment is, and drives the rest of the pipeline (cutting the clip, optional trimming, and publishing to Instagram with a caption generated by Reka).

How we built it

We built AgentClipz as a Python pipeline with clear stages. Chat: We use TwitchIO to connect to Twitch and stream chat into an in-memory buffer; our analytics module computes message rate, top keywords matched with those trending (that we scraped with Yutori), and VADER sentiment and exposes a unified JSON snapshot via get_chat_metrics(). Emotion + speech: We extract audio with ffmpeg and send it to the Modulate API (batch or streaming) for transcription and emotion labels; we get back utterances with timestamps and emotions. Clip detection: We implemented a sliding-window detector that triggers on high-energy emotions (e.g. Excited, Surprised) and/or an overall emotion score threshold; we merge nearby candidates and enforce min/max clip length. Cutting: We use ffmpeg again to cut segments from the source video (with optional subtitle and emotion overlays). The caption is then generated with Reka, based on the content of the video to have a fully posted viral video for Instagram.

Challenges we ran into

One challenge we faced was audio detection. At certain points when our program would recognize something as clip worthy, the project would fully cut the audio out from the stream. When the audio would work, clips would not be made.

Accomplishments that we're proud of

We are proud of making something both complex and extremely fun. Utilizing both Twitch bots and AI agents to generate viral clips proved to allow us to be light-hearted during the design process, paving the way in making this Hackathon a memory to not be forgotten.

What we learned

We learned to work with agents across multiple tasks, whether that be web scraping for trends, Modulate to detect the emotion worthy of a viral clip, or Reka to generate captions as the cherry on top. This on top of a real-time twitch chat detection bot, and the Instagram API, we learned to deal with social medias and applications that many young people often interact with in everyday life.

What's next for AgentClipz

We want to make this work across many other social media platforms (Tiktok, Youtube, ect..), allowing small creators to have the ability of growing across all platforms. We also want to better fine-tune the definition of a "viral-clip," more weight derived from the trending keywords and content we scrape from Yutori.