Skip to content

desenyon/collective

Repository files navigation

Collective

Collective is the organizational memory infrastructure layer for AI software engineering. It lets Codex, Claude Code, Cursor, OpenCode, GitHub Copilot agents, and internal agents reuse verified engineering context instead of repeatedly rediscovering the same repository.

Collective is not a chatbot and not a generic RAG app. The v0.2.0 reset builds the local-first vertical slice: Rust CLI -> Rust daemon -> local memory graph -> context query -> memory write -> dashboard display.

Architecture

Codex / Claude Code / Cursor / OpenCode
        ↓
CLI, MCP tools, or local HTTP
        ↓
Rust local daemon on 127.0.0.1:8787
        ↓
Local memory graph and retrieval pruner
        ↓
Next.js dashboard shell
        ↓
Go cloud API fallback, Neo4j, Qdrant, Postgres, ClickHouse later

Tech Stack

  • Rust: collective-core, collective-daemon, collective-cli, collective-tui, collective-mcp
  • Go: local cloud API in apps/api-go using ConnectRPC plus REST-compatible /v1 endpoints
  • Next.js App Router dashboard files in apps/web
  • Vercel deployment configuration for the dashboard and native Go API Function
  • Clerk launch authentication with local no-secret fallback
  • Local fallback storage in Rust data structures plus .collective-local/cache.sqlite for linked repos, recent context packs, daemon settings, hash-only local auth tokens, and offline writes
  • Docker Compose definitions for Postgres, Redis, Neo4j, Qdrant, and ClickHouse
  • Clerk and WorkOS-compatible auth abstraction documented for cloud mode

Local Setup

pnpm install
pnpm test
go test ./...
pnpm build

The Rust vertical slice does not require external secrets.

Running the Daemon

cargo run -p collective-cli -- daemon
cargo run -p collective-daemon

The daemon listens on http://127.0.0.1:8787 and exposes:

  • GET /health
  • GET /status
  • GET /metrics
  • POST /context/query
  • POST /context/write
  • POST /session/start
  • POST /session/end
  • POST /repo/link
  • POST /repo/sync
  • POST /api-keys
  • DELETE /api-keys/:keyId
  • GET /graph/path

The daemon also exposes local cloud-fallback routes under /v1, including /v1/context/query, /v1/context/write, /v1/agent/session/start, /v1/repo/sync, /v1/orgs, /v1/orgs/:orgId/dashboard, /v1/orgs/:orgId/graph, /v1/orgs/:orgId/activity, /v1/orgs/:orgId/memories, /v1/orgs/:orgId/context-packs, /v1/orgs/:orgId/repositories, /v1/orgs/:orgId/api-keys, /v1/orgs/:orgId/members, /v1/api-keys, and /v1/audit.

CLI

cargo run -p collective-cli -- demo seed
cargo run -p collective-cli -- init
cargo run -p collective-cli -- login --token ck_local_demo_token
cargo run -p collective-cli -- repo link ./demo/demo-saas
cargo run -p collective-cli -- ask "Fix enterprise users being logged out randomly"
cargo run -p collective-cli -- write "Fixed enterprise logout bug by updating middleware TTL logic."
cargo run -p collective-cli -- graph "enterprise middleware"
cargo run -p collective-cli -- session start --agent codex
cargo run -p collective-cli -- dashboard
cargo run -p collective-cli -- daemon --check
cargo run -p collective-cli -- tui

TUI

cargo run -p collective-tui
COLLECTIVE_CACHE_PATH=.collective-local/cache.sqlite cargo run -p collective-tui

The TUI snapshot shows daemon status, linked repos, cached active sessions, cached recent context queries, queued memory writes, graph path, context pack preview, sync queue count, and logs.

MCP and Agent Integrations

cargo run -p collective-mcp

The MCP binary supports a local stdio JSON-RPC fallback. With no stdin it prints the manifest. With JSON-RPC stdin it supports initialize, tools/list, and tools/call for:

  • collective_query_context
  • collective_write_memory
  • collective_get_graph_path
  • collective_get_tests
  • collective_get_recent_decisions
  • collective_start_session
  • collective_end_session

Integration instructions live in docs/agent-integrations.

Example:

printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"collective_query_context","arguments":{"task":"Fix enterprise users being logged out randomly"}}}' \
  | cargo run -p collective-mcp

Demo

cargo run -p collective-cli -- demo seed
cargo run -p collective-daemon
cargo run -p collective-cli -- ask "Fix enterprise users being logged out randomly"

Demo source is in demo/demo-saas. The true cause is source-backed: PR #18 changed enterprise TTL to 12 hours, but middleware still uses the default 2 hour TTL.

Dashboard Data

The web API routes and dashboard pages try the local daemon first through COLLECTIVE_DAEMON_URL or http://127.0.0.1:8787, then fall back to local demo data when the daemon is unavailable. The org list, org overview, graph, repos, chat, sessions, context packs, memories, savings, settings, audit, API keys, and members pages render daemon data when available and fall back to bundled local demo payloads.

Production Deployment: Vercel and Clerk

The dashboard is prepared for Vercel with Clerk as the launch auth provider.

pnpm install --frozen-lockfile
pnpm --dir apps/web build

Create a Vercel project from the repository and use these settings:

  • Framework: Next.js
  • Install command: pnpm install --frozen-lockfile
  • Build command: pnpm --dir apps/web build
  • Output directory: apps/web/.next

Required Vercel environment variables:

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_live_...
CLERK_SECRET_KEY=sk_live_...
NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in
NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up
NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL=/app/demo-org
NEXT_PUBLIC_CLERK_SIGN_UP_FALLBACK_REDIRECT_URL=/app/demo-org
NEXT_PUBLIC_APP_URL=https://your-domain.example
COLLECTIVE_API_URL=https://your-domain.example
COLLECTIVE_DAEMON_URL=http://127.0.0.1:8787

Clerk is wired through apps/web/app/layout.tsx, apps/web/middleware.ts, /sign-in, and /sign-up. When Clerk keys are not present, the app stays runnable in local fallback mode and opens the demo organization without external secrets.

The Go API is wired as a native Vercel Go Function in api/collective.go. The root vercel.json rewrites /health, /v1/*, and /collective.v1.CollectiveAgentService/QueryContext into that function. When COLLECTIVE_API_URL is not set, dashboard server code can fall back to https://$VERCEL_URL after the local daemon probe fails.

Vercel CLI workflow:

pnpm dlx vercel@latest link --yes --project collective --scope <team>
pnpm dlx vercel@latest pull --yes --environment=preview --scope <team>
pnpm dlx vercel@latest build
pnpm dlx vercel@latest deploy --scope <team>

Preview deployments may be protected by Vercel Authentication. Use vercel curl or a Vercel protection bypass token to smoke-test protected preview URLs.

API Example

curl -sS http://127.0.0.1:8787/health
curl -sS -X POST http://127.0.0.1:8787/context/query \
  -d '{"orgId":"org_demo","repoId":"repo_demo_saas","agent":"codex","task":"Fix enterprise users being logged out randomly","mode":"debug","maxTokens":1200}'
curl -sS -X POST http://127.0.0.1:8787/v1/context/query \
  -d '{"orgId":"org_demo","repoId":"repo_demo_saas","agent":"codex","task":"Fix enterprise users being logged out randomly","mode":"debug","maxTokens":1200}'

Running the Go Cloud API

COLLECTIVE_API_ADDR=127.0.0.1:8080 go run ./apps/api-go
go test ./...

The Go API exposes the launch /v1 cloud shape plus a ConnectRPC unary handler at /collective.v1.CollectiveAgentService/QueryContext.

Security Model

  • Every domain object is org scoped.
  • Roles are OWNER, ADMIN, MEMBER, and VIEWER.
  • The local daemon enforces role or API-key scope checks on org-scoped context, memory, session, repo, graph, and status routes.
  • collective login --token ... stores only a hash and short prefix in .collective-local/cache.sqlite; raw local tokens are not stored or printed by the CLI.
  • API key hashes use a one-way hash prefix and raw keys are not stored.
  • API keys can be created by an owner/admin and revoked; revoked keys are rejected on subsequent requests.
  • Scopes include context:read, context:write, repo:sync, repo:ingest, graph:read, metrics:read, and admin.
  • Every daemon response includes a request ID.
  • Audit events are recorded for context queries, memory writes, and sessions.
  • GET /metrics returns local structured request logs and the AGENTS-required metric names for query latency, write latency, graph traversal, retrieval, sync, sessions, context pack size, memory reuse, and errors.

Sustainability

Token savings are exact:

tokens_saved = raw_tokens_estimate - optimized_tokens

Environmental values are approximations based on configurable conversion factors. Collective tracks token reduction exactly, then estimates downstream energy, water, and carbon savings.

Known Limitations

  • Neo4j, Qdrant, ClickHouse, NATS, and Temporal are Docker-defined but not integrated into the local slice.
  • The dashboard builds and runs locally with Next.js; graph and list-page data are daemon-first locally but not streamed from Neo4j/Qdrant.
  • Clerk and Vercel are configured in code, but deployment requires real Vercel and Clerk projects plus production environment variables.
  • The Go cloud API is configured as a Vercel Go Function, but production deployment still needs the Vercel project and environment variables.
  • Vercel preview deployment protection can block unauthenticated curl requests; production domain checks should be repeated after aliases and protection settings are finalized.
  • TUI uses a local snapshot instead of full ratatui/crossterm rendering until external crates are installed.
  • The daemon uses a Rust standard-library HTTP fallback instead of Axum/Tokio until dependency installation is enabled.
  • Offline memory writes are queued and drained in local sync fallback; they do not yet push to the Go cloud API.

Roadmap

  1. Replace std HTTP fallback with Axum/Tokio.
  2. Connect the daemon offline queue to real cloud retry responses.
  3. Persist full daemon graph/session state through SQLite.
  4. Sync local graph into Neo4j and Qdrant.
  5. Feed the interactive graph explorer from live graph APIs instead of demo data.

About

Collective is the organizational memory infrastructure layer for AI software engineering. It lets Codex, Claude Code, Cursor, OpenCode, GitHub Copilot agents, and internal agents reuse verified engineering context instead of repeatedly rediscovering the same repository.

Resources

Stars

Watchers

Forks

Contributors