Skip to content
This repository was archived by the owner on Oct 26, 2025. It is now read-only.

skatari172/Hermes

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

62 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Hermes – AI Cultural Companion

An intelligent travel companion that uses AI to understand and document cultural experiences through image analysis, location-aware conversations, and automatic journaling.

Tech Stack

Frontend: React Native (Expo), TypeScript, Firebase, Axios
Backend: FastAPI, Python, Gemini 2.5 Flash
AI: Google Gemini Vision API
Storage: Firebase Firestore + Firebase Storage
Location: OpenStreetMap (free reverse geocoding) + Wikipedia geosearch (free)

How It Works

Core Flow

  1. User uploads photo → Backend analyzes with Gemini Vision
  2. Location detected → OpenStreetMap reverse geocoding + Wikipedia nearby search
  3. Context built → Combines image analysis + location + landmarks
  4. AI response → Location-aware cultural insights with recommendations
  5. Storage → Saves to Firestore, creates journal entries

Architecture

Frontend (React Native) → Backend (FastAPI) ├─→ Gemini AI (Vision + Chat) ├─→ OpenStreetMap (Location) ├─→ Wikipedia (Nearby Attractions) └─→ Firebase (Storage + Database)

Image Processing Pipeline: Photo Upload → Perception Analysis (Gemini Vision) ↓ Geo Detection (OpenStreetMap + Wikipedia) ↓ Context Building (Merge data) ↓ AI Response (Location-aware insights) ↓ Storage (Firestore + Journal)

Agent Layer

Key Features

  • Location-Aware: Detects city/country, fetches nearby Wikipedia attractions
  • Translation Priority: Translates foreign text in images (signs, menus, etc.)
  • Smart Responses: Initial photos get detailed analysis (4 sections), follow-ups are ultra-concise (20 words max)
  • Auto Journaling: Conversations automatically saved as travel journals

Quick Start

# Backend
cd backend && python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 8000 --reload

# Frontend  
cd frontend && npm install
npx expo start

Required .env variables:

GEMINI_API_KEY=your_key
FIREBASE_PROJECT_ID=your_project
FIREBASE_API_KEY=your_api_key
FIREBASE_AUTH_DOMAIN=your_domain
FIREBASE_STORAGE_BUCKET=your_bucket
FIREBASE_MESSAGING_SENDER_ID=your_id
FIREBASE_APP_ID=your_id

Core Components

Frontend: app/screens/HomeScreen.tsx - Photo upload + chat interface
Backend:

  • main.py - FastAPI endpoints
  • utils/perception_utils.py - Image analysis
  • utils/geo_api_utils.py - Location detection
  • utils/response_utils.py - AI response generation
  • services/db_service.py - Firestore operations

API Endpoints

  • POST /api/image/process - Upload image, get AI analysis
  • POST /api/chat - Send chat message
  • GET /api/journal - Get user journals

Docs: http://localhost:8000/docs Contributors:

Sujan Katari: Multi-Agent Orchestration + Backend
Nick Gonzalez: Database + Fullstack
Vignesh Sundararajan: Fullstack + Voice Functionality
Ayaan Khan: Image Processing + Frontend (Camera Functionality)

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors