Hermes AI

Inspiration

Everyone has experienced that sinking feeling when losing something valuable. Traditional lost and found systems are fundamentally flawed: they're inefficient, lack proper verification, and expose inventory to false claims. We built Hermes AI to solve this with a blind vault system that keeps inventory completely hidden until verified matches are confirmed. This prevents fraud while making item recovery seamless and secure.

What it does

Hermes AI is an AI-powered lost and found platform that eliminates manual searching and prevents fraud through a unique zero-knowledge matching system. Users who have lost items can submit inquiries with photos and text descriptions. The system automatically analyzes submissions using Google Gemini Vision API, extracting detailed metadata including category, brand, colours, notable identifiers, and condition. The AI then searches against a hidden catalog that users never see, preventing false claims. When a potential match is found with sufficient confidence, assistants review it in a verification cockpit that compares the user's claim against the actual inventory. Only after assistant approval do users see match details. The system includes multi-layered fraud prevention including rate limiting, duplicate detection, image verification, ownership verification questions, and confidence scoring. All status updates happen in real-time, so users always know where their inquiry stands in the workflow from submission to resolution.

How we built it

We built Hermes AI using Next.js 16 with the App Router and React 19, all written in TypeScript for type safety. The frontend uses Tailwind CSS for a modern, responsive interface with real-time updates powered by Supabase subscriptions. The backend uses Supabase for PostgreSQL database, authentication, and image storage. Google Gemini Vision API automatically analyzes images when items are uploaded, extracting structured metadata that feeds into our matching algorithm. The matching engine uses a weighted multi-factor scoring system that considers category match, color similarity, condition reports, location proximity, and notable identifiers. We implemented role-based access control where assistants have exclusive access to the hidden catalog, while regular users can only see match metadata until verification is complete. The system includes serverless API routes for processing inquiries, matching items, and managing the verification workflow. Real-time synchronization ensures all users see status updates instantly without page refreshes.

Challenges we ran into

The most significant challenge was implementing real-time synchronization across all components. We needed to ensure that status updates, new matches, and assistant actions were reflected immediately for all users without polling or manual refreshes. This required careful implementation of Supabase subscriptions and managing state across multiple React contexts. Another major challenge was building the complex status lifecycle management system that handles transitions from submitted to under review to matched to resolved, ensuring data consistency at each stage. The AI processing pipeline also presented challenges in handling image uploads, generating signed URLs for storage, calling the Gemini API reliably, and storing metadata correctly. We had to implement robust error handling and retry logic for the AI integration. The access control system was tricky because we needed assistants to access the full catalog while keeping it completely hidden from regular users, requiring careful permission checks at every database query. Finally, balancing the matching algorithm's accuracy with performance was challenging, especially when dealing with large catalogs and multiple match candidates.

Accomplishments that we're proud of

We're most proud of creating the Blind Vault architecture, a revolutionary approach to preventing false claims by keeping inventory completely hidden until verified matches. This fundamentally changes how lost and found systems work. We successfully integrated Google Gemini Vision API seamlessly into the background, making AI analysis invisible to users while providing accurate metadata extraction. We built a production-ready system with all fifteen required features fully implemented, tested, and connected, including inquiry submission, automatic matching, assistant review workflows, follow-up questions, and comprehensive fraud prevention. Our multi-layered security system includes six different fraud prevention mechanisms working together. The smart matching system that automatically generates follow-up questions when too many matches exist demonstrates sophisticated problem-solving. Finally, we created a real-time experience where users see live status updates without any page refreshes, making the system feel modern and responsive.

What we learned

We learned how to successfully integrate Google Gemini Vision API for automatic image analysis, including handling API responses, parsing JSON from AI outputs, and storing structured metadata. We gained deep understanding of building a zero-knowledge security architecture that prevents fraud while maintaining usability. Implementing real-time systems with Supabase subscriptions taught us about managing live data synchronization and handling edge cases in concurrent updates. We learned to manage complex state across multiple React contexts while maintaining clean code architecture. Building multi-role workflows with proper access controls required careful planning of database permissions and frontend authorization checks. Most importantly, we learned how to build a complete, production-ready system within a hackathon timeframe by focusing on core features and implementing them thoroughly rather than adding many incomplete features.

What's next for Hermes AI

We plan to build a mobile app for iOS and Android to make item recovery accessible on the go. We'll add push notifications for match alerts so users don't have to constantly check their inquiry status. We want to enhance the AI analysis with more detailed extraction capabilities and improve the matching algorithm's accuracy. Multi-language support will make Hermes AI accessible to international users. For long-term expansion, we envision scaling to cities with municipal partnerships, integrating with airports for travel-related losses, and forming partnerships with transit systems including buses, trains, and subways. Corporate solutions for office campuses and event venues could provide temporary lost and found services during large gatherings. We're also considering building an API for third-party integrations, allowing other platforms to leverage our matching technology. The ultimate goal is to make Hermes AI the standard for secure, AI-powered lost and found services across all sectors where items are frequently lost.