AI-powered legal intelligence platform — Processes multimodal evidence through domain-specialized Gemini 3 agents with full reasoning transparency.

Built for the Gemini 3 Global Hackathon 2026.

To try Holmes without setting up locally, use the deployed production instance:

Upload the demo case files to a case and run the analysis pipeline from the Command Center to see the full agent flow, knowledge graph, timeline, geospatial map, and verdict dashboard.

Legal professionals and investigators spend 60%+ of their time manually reviewing evidence across scattered formats — PDFs, video depositions, audio recordings, financial documents, images. They must cross-reference every piece to find connections, contradictions, and gaps. Current AI tools suffer from hallucination and operate as black boxes, making them unusable for legal work that demands transparency, auditability, and exact source citations.

Holmes processes any combination of evidence types through a pipeline of 10 domain-specialized AI agents, then surfaces connections, contradictions, and gaps — while showing exactly how it reached those conclusions through Agent Flow, a real-time visualization of AI reasoning. Every factual claim traces back to its exact source location.

• Demo
• Key Features
• Architecture
• Agent Pipeline — Deep Dive
• Gemini 3 Integration
• Frontend Visualizations
• Real-Time SSE Streaming
• AI-Powered Redaction
• Investigator's Notebook
• Database Schema
• Tech Stack
• Project Structure
• Quick Start
• Make Commands
• Environment Variables
• Deployment
• License

┌──────────────────────────────────────────────────────────────────┐
│                          Frontend                                 │
│  Next.js 16 · React 19 · React Flow · D3.js · Google Maps       │
│  Agent Flow | Knowledge Graph | Timeline | Map | Verdict | Chat  │
└────────────────────────────┬─────────────────────────────────────┘
                             │ SSE (real-time streaming)
┌────────────────────────────▼─────────────────────────────────────┐
│                          Backend                                  │
│  FastAPI · Google ADK · 10-Agent Pipeline · SSE Publisher         │
│                                                                   │
│  Triage → Orchestrator → Domain Agents (parallel) → Strategy     │
│  → HITL Confirmation → KG Builder → Synthesis → Geospatial       │
└────────────────────────────┬─────────────────────────────────────┘
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   PostgreSQL    │ │  Cloud Storage  │ │    Gemini 3     │
│  Sessions, KG,  │ │   Evidence      │ │  Pro (reasoning)│
│  Findings,      │ │   Files         │ │  Flash (triage) │
│  Synthesis      │ │                 │ │  File API (2GB) │
└─────────────────┘ └─────────────────┘ └─────────────────┘

• Stage-isolated sessions — Each pipeline stage creates a fresh ADK session to prevent multimodal file content from bloating downstream contexts. Inter-stage data flows via database, not session state.
• Config-driven agents — Domain agents share a generic DomainAgentRunner[OutputT] template, eliminating ~1200 lines of duplicated run/retry/fallback logic.
• Pro-to-Flash fallback — Domain agents attempt Gemini 3 Pro first, falling back to Flash on failure for graceful degradation.
• Deterministic session IDs — SHA-256(case_id:workflow_id:stage) enables idempotent reruns.
• Pipeline orchestration via Python — NOT a single ADK SequentialAgent. Each stage is an independent runner invocation, with parallel execution within stages via asyncio.gather.

The pipeline processes evidence through 8 stages, each running in a fresh ADK session:

Classifies every file by domain relevance (Financial 0-100, Legal 0-100, Evidence 0-100, Strategy 0-100), extracts entities (people, organizations, dates, locations, amounts, legal terms), generates summaries, and assigns complexity tiers. Retry logic creates a fresh session on parse failure to avoid polluted context.

Output: TriageOutput → domain scores, entity lists, summaries, complexity tier, file groupings.

Files: backend/app/agents/triage.py, backend/app/agents/prompts/triage.py

Routes files to domain agents based on triage scores. Produces routing decisions with confidence scores, file groups for joint analysis, and parallel/sequential execution plans. Triggers Human-in-the-Loop confirmation when per-agent confidence falls below thresholds (Financial: 50, Legal: 50, Evidence: 20, Strategy: 40).

Output: OrchestratorOutput → routing decisions, file groups, parallel agents, sequential agents.

Files: backend/app/agents/orchestrator.py, backend/app/agents/prompts/orchestrator.py

Each agent type runs one instance per file group concurrently via asyncio.gather. Each instance creates its own database session (async sessions are not thread-safe). All agents use the config-driven DomainAgentRunner[OutputT] template.

Pro-to-Flash Fallback: If Pro fails after retries, the runner automatically attempts Flash and emits an agent-fallback SSE event.

Output: DomainAgentOutput → findings with span-level citations, extracted entities, quality assessments (Evidence agent).

Files: backend/app/agents/domain_agent_runner.py, backend/app/agents/financial.py, backend/app/agents/legal.py, backend/app/agents/evidence.py

Runs after parallel domain agents complete. Receives text summaries of all domain findings (not raw files) plus any files explicitly routed to strategy. Produces strategic recommendations and investigation direction analysis.

Files: backend/app/agents/strategy.py

Two HITL checkpoints:

1. Routing HITL — Before domain agents run, if orchestrator confidence is below threshold for any agent type. Batch confirmation via SSE.
2. Finding HITL — After domain agents, for findings with confidence < 40.

Confirmation requests are sent as SSE events. The pipeline awaits user response with a configurable timeout (default: 1 hour).

Files: backend/app/services/confirmation.py, backend/app/services/pipeline.py

An LLM agent (not programmatic) reads all domain findings holistically and produces a curated knowledge graph with:

• Entities — PERSON, ORGANIZATION, LOCATION, EVENT, ASSET, FINANCIAL_ENTITY, COMMUNICATION, DOCUMENT — with aliases, descriptions, domain tags, and confidence scores
• Relationships — Semantic typed labels (e.g., employed_by, transferred_funds_to, owns) with evidence excerpts and temporal context

Uses clear-and-rebuild pattern: deletes old KG data for the workflow, then writes curated entities and relationships. Lenient parsing with savepoints — malformed items are skipped, not fatal.

Files: backend/app/agents/kg_builder.py, backend/app/agents/prompts/kg_builder.py

Cross-references all domain findings and knowledge graph data to produce:

Writes to 7 database tables. This is where Gemini 3's 1M context window is critical — the agent sees all findings, all entities, and all relationships in a single call.

Files: backend/app/agents/synthesis.py, backend/app/agents/prompts/synthesis.py

Triggered when synthesis results contain location data. Extracts locations, geocodes via Google Maps API, identifies movement patterns, and flags locations for satellite imagery analysis.

Files: backend/app/agents/geospatial.py, backend/app/services/geocoding_service.py

Not part of the automatic pipeline. Answers user questions via four specialized tools:

Each tool factory captures case_id via closure and creates independent database sessions per call. The agent's system prompt includes the case summary for immediate context.

Files: backend/app/agents/chat_tools.py, backend/app/services/chat_service.py

All agents use ThinkingConfig at HIGH level via ADK's BuiltInPlanner:

# backend/app/agents/base.py
def create_thinking_planner(level: str = "high") -> BuiltInPlanner:
    return BuiltInPlanner(
        thinking_config=ThinkingConfig(
            thinking_level=ThinkingLevel.HIGH,
            include_thoughts=True,
        )
    )

include_thoughts=True captures thinking traces which are stored in agent_executions.thinking_traces (JSONB) and streamed to the frontend via SSE for full transparency.

Gemini 3 Pro processes PDFs, videos, audio, and images natively — no transcription, OCR, or preprocessing pipelines. Domain agents receive raw evidence files directly:

# backend/app/services/adk_service.py
async def build_domain_agent_content(files, gcs_bucket, prompt):
    parts = [types.Part.from_text(text=prompt)]
    for f in files:
        # Force File API for video/audio regardless of size
        if f.mime_type.startswith(("video/", "audio/")):
            file_part = await prepare_file_via_api(...)
        else:
            file_part = await prepare_file_for_agent(f, gcs_bucket)
        parts.append(file_part)
    return types.Content(role="user", parts=parts)

File API uploads include async polling with backoff until the file state is ACTIVE, with a configurable timeout (default 5 minutes).

# backend/app/agents/factory.py (Financial example)
generate_content_config=types.GenerateContentConfig(
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_HIGH,
)

The stage-isolated architecture strategically uses the 1M context window:

• Triage — Processes ALL files in one call (50-200K tokens typical)
• Domain Agents — File-group subsets (100-300K tokens per instance)
• Synthesis — All findings + all KG data in one call (200-500K tokens) — this is where 1M context is critical for holistic cross-domain analysis

Context is NOT accumulated across stages. Each stage gets a fresh session, preventing "context explosion" from multimodal content.

Real-time visualization of the agent pipeline built with @xyflow/react + dagre layout:

• Hierarchical decision tree showing agent nodes with status indicators (running/complete/error)
• Animated edges between decision and file-group nodes
• Click any node to open the details sidebar: model name, input context, tool calls, output findings, token usage (input/output/thoughts), duration
• Execution timeline (Gantt chart) showing agent timing overlap
• Full thinking traces displayed untruncated
• Auto-fitting viewport with smooth animations on node addition

Files: frontend/src/components/CommandCenter/

D3.js force-directed graph with Epstein-inspired layout:

• 5 forces — link, charge (-400), center, collision, radial (connection-count gravity well)
• Node sizing — Sqrt-scaled radius by connection count (d3.scalePow().exponent(0.5), 5-100px)
• Domain colors — Person (orange), Organization (green), Location (blue), Event (amber)
• Left panel — Filters, entity search, keyword filter, domain/type toggles, graph stats
• Right panel — Entity timeline showing chronological relationships when a node is selected
• Interactions — Click to select, drag to move, scroll to zoom, pan
• Zoom/pan with d3.zoom() (scale extent [0.01, 10])

Files: frontend/src/components/knowledge-graph/, frontend/src/hooks/useGraphSimulation.ts

• Horizontal timeline with day/week/month/year zoom levels
• Events grouped by zoom level, colored by layer (evidence/legal/strategy/financial)
• Click event to open detail modal with source citations
• Real-time updates via SSE for event creation/deletion

Files: frontend/src/components/Timeline/

Scrollable multi-section intelligence dashboard:

1. Summary & Verdict — Case conclusion with confidence label (Conclusive/Substantial/Inconclusive)
2. Key Findings — 2-column grid ranked by confidence
3. Hypotheses — Cards with supporting/contradicting evidence counts
4. Contradictions — Side-by-side conflicting claims with severity badges
5. Evidence Gaps — Missing information with priority and suggested actions
6. Investigation Tasks — Follow-up actions derived from synthesis

Files: frontend/src/components/verdict/

• Google Maps via @vis.gl/react-google-maps
• Landmark markers color-coded by type (crime scene, witness, evidence, suspect)
• Movement paths between locations
• Street View integration
• Location detail panel with geocoding metadata

Files: frontend/src/components/Geospatial/

The pipeline publishes events at every agent lifecycle point via ADK callbacks:

Additional events: state-snapshot (reconnection), confirmation-required (HITL), synthesis-data-ready, processing-complete.

• sse-starlette for FastAPI SSE endpoints
• Heartbeat every 15 seconds (X-Accel-Buffering: no) to prevent Cloud Run timeout
• State snapshot on reconnect — built from AgentExecution records in the database
• In-memory pubsub with per-case subscriber queues (maxsize 100)
• Token delta streaming — input tokens, output tokens, and thought tokens reported per model call

Reconnection with exponential backoff (max 5 attempts, 2s initial delay).

Three redaction agents for censoring sensitive content via natural language prompts:

Gemini 3 Flash reads the full PDF, identifies content matching the user's prompt (e.g., "Redact all personal names and phone numbers"), and returns structured RedactionTarget objects with exact text, page number, and surrounding context. PyMuPDF then applies black-box redactions at the identified locations.

Files: backend/app/agents/redaction.py

AI-powered segmentation identifies regions to censor (faces, license plates, etc.) based on natural language prompts. Supports blur and pixelate methods. Returns both the censored image and a visualization showing detected masks.

Files: backend/app/agents/image_redaction.py

Frame-by-frame processing with AI-powered region detection. Supports blur, pixelate, and blackbox methods. Returns censored video with processing logs and frame-level statistics.

Files: backend/app/agents/video_redaction.py

POST /api/cases/{caseId}/files/{fileId}/redact          # PDF redaction
POST /api/cases/{caseId}/files/{fileId}/redact-image     # Image redaction
POST /api/cases/{caseId}/files/{fileId}/redact-video     # Video redaction

A case-scoped note-taking system ("Sherlock's Diary") for investigators:

• Voice notes — Browser MediaRecorder API captures audio, stored as blobs
• Text notes — Rich text editor for written observations
• Case-linked — All notes tied to the active case for context
• CRUD API — Create, list, update, delete notes per case

Files: frontend/src/components/notebook/Notebook.tsx, backend/app/api/notes.py, backend/app/models/note.py

PostgreSQL 17 with JSONB for flexible metadata. Key tables:

All tables use UUIDs, are indexed on case_id, and support JSONB for flexible metadata. Migrations managed by Alembic.

holmes/
├── backend/
│   ├── app/
│   │   ├── agents/              # All 10+ agent implementations
│   │   │   ├── base.py          # Model constants, thinking planner, SSE callbacks
│   │   │   ├── factory.py       # AgentFactory — creates fresh agent instances
│   │   │   ├── triage.py        # Stage 1: File classification
│   │   │   ├── orchestrator.py  # Stage 2: Routing decisions
│   │   │   ├── domain_agent_runner.py  # Generic runner template
│   │   │   ├── financial.py     # Domain: Financial analysis config
│   │   │   ├── legal.py         # Domain: Legal analysis config
│   │   │   ├── evidence.py      # Domain: Evidence/forensic config
│   │   │   ├── strategy.py      # Domain: Strategy analysis (sequential)
│   │   │   ├── kg_builder.py    # Stage 6: LLM-based knowledge graph
│   │   │   ├── synthesis.py     # Stage 7: Cross-domain synthesis
│   │   │   ├── geospatial.py    # Stage 8: Location intelligence
│   │   │   ├── chat_tools.py    # Chat agent's 4 retrieval tools
│   │   │   ├── redaction.py     # PDF redaction agent
│   │   │   ├── image_redaction.py   # Image censorship agent
│   │   │   ├── video_redaction.py   # Video censorship agent
│   │   │   └── prompts/         # System prompts for each agent
│   │   ├── api/                 # FastAPI route handlers
│   │   ├── models/              # SQLAlchemy ORM models
│   │   ├── schemas/             # Pydantic request/response schemas
│   │   └── services/            # Business logic
│   │       ├── pipeline.py      # Main pipeline orchestrator (~1200 lines)
│   │       ├── adk_service.py   # ADK session management, file preparation
│   │       ├── chat_service.py  # Chat agent runner
│   │       └── confirmation.py  # HITL confirmation service
│   ├── alembic/                 # Database migrations
│   ├── Dockerfile
│   └── pyproject.toml
├── frontend/
│   ├── src/
│   │   ├── components/
│   │   │   ├── CommandCenter/   # Agent Flow canvas + detail sidebar
│   │   │   ├── knowledge-graph/ # D3.js graph + filter/timeline panels
│   │   │   ├── Timeline/        # Chronological event view
│   │   │   ├── verdict/         # Synthesis dashboard
│   │   │   ├── Geospatial/      # Google Maps view
│   │   │   ├── library/         # Evidence file management + redaction
│   │   │   ├── notebook/        # Investigator's notebook
│   │   │   ├── source-viewer/   # PDF/video/audio/image viewers
│   │   │   └── app/             # Layout, sidebar, chatbot
│   │   ├── hooks/               # React hooks (SSE, graph simulation, filters)
│   │   └── lib/                 # API clients, utilities, citation parsing
│   ├── Dockerfile
│   └── package.json
├── packages/types/              # Shared TypeScript types (auto-generated from OpenAPI)
├── .github/workflows/           # CI/CD pipeline
├── docker-compose.yml           # Local PostgreSQL + Adminer
├── Makefile                     # Development commands
├── .env.example                 # Environment variable template
└── .planning/                   # Project roadmap and architecture docs

• Node.js ≥20 and Bun (frontend)
• Python ≥3.12 and uv (backend)
• Docker (local PostgreSQL)
• Google API Key from AI Studio
• Google Maps API Key (for geospatial features)

# 1. Clone and install dependencies
git clone <repo-url> && cd holmes
make install

# 2. Configure environment
cp .env.example backend/.env
# Edit backend/.env — set GOOGLE_API_KEY and GOOGLE_MAPS_API_KEY

# 3. Start database and run migrations
make dev-db
make migrate

# 4. Start services (separate terminals)
make dev-backend    # → http://localhost:8080
make dev-frontend   # → http://localhost:3000

# Database admin UI
make adminer        # → http://localhost:8081

# Generate TypeScript types from backend OpenAPI
make generate-types

# Run compliance checks (lint + format + types)
make check

Create backend/.env from .env.example:

• Cloud Run — Separate services for frontend and backend (europe-west3)
• Cloud SQL — PostgreSQL 17 managed instance
• Cloud Storage — Evidence file buckets (per environment)
• Artifact Registry — Docker images
• Secret Manager — Environment variables
• Workload Identity Federation — Keyless GitHub Actions → GCP authentication

GitHub Actions workflow on push to main:

1. Type Generation — make generate-types, fail if uncommitted changes
2. Backend Build — Ruff lint, mypy type check, Docker build → Artifact Registry
3. Frontend Build — Next.js build, Docker build → Artifact Registry
4. Deploy — Alembic migrations, Cloud Run deployment for both services

Apache License Version 2.0