24/7 autonomous SEO and Generative Engine Optimization platform. Track keyword rankings, find content gaps, and optimize your visibility in AI-powered search engines like ChatGPT, Perplexity, and Gemini.

Crawl is a multi-tenant SaaS application that combines traditional SEO with Generative Engine Optimization (GEO) — making sure your content gets cited by large language models, not just ranked on Google.

Key capabilities:

• Keyword Ranking Tracker — Pulls live SERP data via DataForSEO, synced with Google Search Console
• GEO / LLM Visibility — Checks how often your site is cited by ChatGPT, Perplexity, and Gemini
• AI SEO Audits — Autonomous LangGraph agent runs full SEO + GEO audits on demand
• AI Blog Generation — Generates and edits SEO-optimized blog posts with a rich TipTap editor
• Google Integrations — Connects to Google Search Console and Google Analytics 4
• Sandbox Mode — Time-limited demo environments for prospecting

genai-genesis/
├── crawl/                        # Turbo monorepo (main project)
│   ├── apps/
│   │   └── web/                  # Next.js 14 frontend
│   │       ├── app/              # App Router pages
│   │       │   ├── dashboard/    # Core app (rankings, GEO, AI blogs, settings)
│   │       │   ├── sandbox/      # Demo/prospect environment
│   │       │   ├── auth/login/   # Authentication
│   │       │   ├── blog/         # Public blog
│   │       │   └── onboarding/   # New user onboarding
│   │       ├── components/       # Shared UI components (Radix-based)
│   │       └── lib/              # Supabase, Gemini, Google OAuth, encryption
│   ├── services/
│   │   └── agent/                # Python FastAPI AI agent service
│   │       ├── main.py           # FastAPI entrypoint
│   │       ├── graphs/
│   │       │   ├── orchestrator.py   # LangGraph orchestrator
│   │       │   ├── seo_graph.py      # SEO audit subgraph
│   │       │   └── geo_graph.py      # GEO/LLM visibility subgraph
│   │       ├── clients/
│   │       │   └── dataforseo.py     # DataForSEO API client
│   │       ├── nodes/            # LangGraph node implementations
│   │       └── tools/            # Agent tool definitions
│   ├── docker/
│   │   ├── docker-compose.yml
│   │   └── docker-compose.sandbox.yml
│   ├── turbo.json
│   └── vercel.json
└── openclawseo/                  # Alternate branding / earlier iteration

• Google Search Console — keyword impressions, clicks, positions
• Google Analytics 4 — traffic and conversion data
• DataForSEO — SERP rank tracking
• Perplexity API — LLM citation checking
• Framer — design/content platform integration
• Google Gemini — AI audit generation and blog writing

Key tables:

• Node.js 20+
• Python 3.12+
• Docker & Docker Compose
• A Supabase project
• A Google Cloud project (for OAuth + GSC + GA4)
• DataForSEO account
• Google Gemini API key
• Upstash Redis instance

git clone <repo-url>
cd genai-genesis/crawl
npm install

Create apps/web/.env.local:

# Supabase
NEXT_PUBLIC_SUPABASE_URL=https://<project>.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=<anon-key>
SUPABASE_SERVICE_ROLE_KEY=<service-role-key>

# Google OAuth (Search Console + GA4)
GOOGLE_CLIENT_ID=<client-id>
GOOGLE_CLIENT_SECRET=<client-secret>
GOOGLE_REDIRECT_URI=http://localhost:3000/api/auth/google/callback
GA4_REDIRECT_URI=http://localhost:3000/api/auth/ga4/callback

# AI
GEMINI_API_KEY=<gemini-api-key>
PERPLEXITY_API_KEY=<optional>

# Framer
FRAMER_API_KEY=<framer-api-key>
FRAMER_PROJECT_ID=<framer-project-id>

# Upstash Redis
UPSTASH_REDIS_REST_URL=<url>
UPSTASH_REDIS_REST_TOKEN=<token>

# Security
ENCRYPTION_SECRET=<32-char-secret>

# Agent Service
AGENT_SERVICE_URL=http://localhost:8000
AGENT_WEBHOOK_SECRET=<webhook-secret>

# App
NEXT_PUBLIC_APP_URL=http://localhost:3000
NEXT_PUBLIC_SANDBOX_DOMAIN=sandbox.localhost:3000

Create services/agent/.env:

SUPABASE_URL=https://<project>.supabase.co
SUPABASE_SERVICE_ROLE_KEY=<service-role-key>
GEMINI_API_KEY=<gemini-api-key>
DATAFORSEO_API_LOGIN=<login>
DATAFORSEO_API_PASSWORD=<password>
PERPLEXITY_API_KEY=<optional>
AGENT_WEBHOOK_SECRET=<same-as-above>
NEXT_PUBLIC_APP_URL=http://localhost:3000

cd services/agent
pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 8000 --reload

Or via Docker:

cd docker
docker-compose up --build

cd crawl
npm run dev

The app will be available at http://localhost:3000.

From crawl/ (runs all workspaces via Turbo):

npm run dev          # Start all dev servers
npm run build        # Build all apps
npm run lint         # Lint all code
npm run type-check   # TypeScript type checking

From crawl/apps/web/:

npm run dev          # Next.js dev server (port 3000)
npm run build        # Production build
npm run start        # Start production server

The Python agent service uses LangGraph to run multi-step SEO and GEO audits asynchronously.

Next.js  →  POST /api/agent/run  →  FastAPI  →  LangGraph Orchestrator
                                                       ├── SEO Graph
                                                       │     ├── Fetch GSC data
                                                       │     ├── Run SERP checks
                                                       │     └── Generate AI recommendations
                                                       └── GEO Graph
                                                             ├── Query Perplexity
                                                             ├── Query Gemini
                                                             └── Check citations
FastAPI  →  POST /api/agent/webhook  →  Next.js  (async result delivery)

Long-running audits are handled via webhooks since they exceed serverless function timeouts.

The project is configured for Vercel deployment:

// vercel.json
{
  "buildCommand": "npm run build",
  "outputDirectory": "apps/web/.next"
}

Push to main to trigger automatic deployment.

cd crawl/docker
docker-compose -f docker-compose.yml up -d

For production, expose the agent service via a public URL (e.g., ngrok tunnel or cloud VM) and set AGENT_SERVICE_URL accordingly in your frontend environment.

Middleware at crawl/apps/web/middleware.ts protects:

• /dashboard/* — requires authenticated session
• /sandbox/control-panel/* — requires authenticated session
• Logged-in users are redirected away from /auth/login

The platform is fully multi-tenant:

• Users belong to one or more Organizations
• Each organization has Sites (tracked websites)
• Roles: admin, member, viewer
• Sandbox environments provide isolated, time-limited demo access for prospects

Private — all rights reserved.