GhOOL

Inspiration

The project was inspired by the need for more efficient video search and analysis tools, specifically for finding relevant video clips and combining them based on specific qualities. We wanted to leverage the power of Twelve Labs' AI to process video content and automate complex video editing tasks like smooth transitions and quality evaluations. The project aims to simplify video content creation by using AI for video content analysis and transition evaluation.

What it does

This project uses Twelve Labs’ API to search and retrieve relevant video clips based on user prompts. It then processes these videos by concatenating them and evaluating the transitions between different videos based on certain quality metrics, such as smoothness. Users can input prompts and retrieve videos that match the desired themes, and the project ranks the video transitions based on predefined metrics like transition quality or visual consistency.

How we built it

We built the project using Python, integrating several powerful libraries:

Twelve Labs API: To search, retrieve, and process video content based on textual prompts.
MoviePy: To edit videos, including concatenation, subclipping, and adding transitions.
FFmpeg: For handling video processing at a more detailed level.
Requests: To communicate with the Twelve Labs API for video analysis, generation, and quality evaluation.

The project starts by searching for video clips based on user prompts and ranking them using the AI's confidence levels. The best clips are concatenated, and their transitions are evaluated using another call to the Twelve Labs API.

Challenges we ran into

API Quotas: Running out of API credits was a challenge that required managing multiple API keys and improving how we utilized API calls efficiently.
Video Processing: Handling video formats and ensuring smooth playback across different devices required fine-tuning the video quality and frame rate. Additionally, ensuring seamless transitions between video clips was difficult due to mismatches in video properties (e.g., frame rates, resolutions).
Synchronization of Video and Audio: Ensuring that concatenated videos had consistent audio was another technical challenge, as missing audio properties like fps caused errors during playback.

Accomplishments that we're proud of

Automated Transition Evaluation: One of the most significant accomplishments was creating a system that automatically evaluates the quality of transitions between two concatenated video clips, based on predefined criteria like smoothness.
Efficient Video Search: The integration with Twelve Labs API allowed for a very efficient search process, retrieving high-quality video clips relevant to the user’s input prompts in a matter of seconds.
Smooth Integration of Tools: Combining MoviePy, Twelve Labs, and Requests into a cohesive pipeline for video processing and evaluation is something we’re particularly proud of.

What we learned

We learned a lot about video processing using MoviePy and the importance of balancing computational costs with output quality. Working with APIs for video search and evaluation also taught us about managing quotas and efficiently handling large datasets. Additionally, we gained deeper insights into video properties such as fps, audio synchronization, and resolution management when working with different media sources.

What's next for GhOOL

Moving forward, we plan to:

Add More Quality Metrics: Improve the video evaluation system by introducing more quality metrics, such as emotional tone matching between video clips or visual style consistency.
Improve User Experience: Add a graphical interface that allows users to search, preview, and download video clips directly without needing to input commands manually.
Optimize API Usage: Further optimize the API usage to manage and distribute requests more efficiently, reducing the chance of running out of API credits.