Skip to content

joshwu108/RC.JS

Repository files navigation

RC.JS

A web-first, source-grounded thinking workspace where PDFs, notes, GitHub content, and Slack exports become a navigable knowledge graph.

RC.JS is inspired by NotebookLM-style research workflows, but rendered as a spatial graph experience: ingest sources, generate concepts/questions/tasks, ask grounded questions, and explore evidence paths.

Table of Contents

Overview

RC.JS turns source material into a graph-shaped workspace for thinking and decision-making:

  • Start from a prompt and create a workspace.
  • Ingest sources (PDFs, notes, GitHub, Slack export text).
  • Build a typed graph (concepts, entities, questions, tasks, insights, clusters).
  • Ask questions and get source-grounded answers with citations and graph highlights.
  • Publish and browse public workspaces; fork public work into private workspaces.

The product direction is organized around two principles from specs/README.md:

  • Centralization: good knowledge should be reusable through a shared public graph.
  • Validation: answers and graph claims should be grounded in explicit source evidence.

Core Capabilities

  • Prompt-first landing experience and workspace flow.
  • Private and public workspaces.
  • Source ingestion pipeline with chunking and extraction.
  • 2D/3D graph visualization for workspace exploration.
  • Ask pipeline with citations and highlight IDs.
  • Public gallery and fork flow.
  • Background job orchestration via Postgres queues.

Current Implementation Status

Status is based on implementation-plan.md, specs/README.md, and domain specs.

  • Implemented:
    • Workspace CRUD and graph/ask APIs.
    • Public gallery, public workspace read, and public fork API.
    • Source endpoints (upload, note, GitHub, Slack import).
    • Ingestion jobs and worker orchestration with inline/background modes.
    • AI pipeline modules for chunking, extraction, embedding retrieval, and grounded answers.
    • Unit/integration tests and Playwright E2E flow scaffolding.
  • Partially implemented or in-progress:
    • Some UX wiring from landing/workspace controls is still being polished in product-experience spec checklists.
    • End-to-end reliability depends on complete local env setup (Supabase + Gemini keys).

Tech Stack

  • Frontend:
    • Next.js 16, React 19, TypeScript
    • Tailwind CSS 4
    • Framer Motion + GSAP
    • react-force-graph-3d + Three.js
    • Zustand
  • Backend:
    • Next.js Route Handlers
    • Supabase (Auth, Postgres, Storage, RLS)
    • pgvector for embeddings
    • pg-boss for background orchestration
  • AI:
    • Google Gemini API via @google/generative-ai
    • Embeddings + structured extraction and grounded answer flows
  • Testing:
    • Vitest + Testing Library
    • Playwright

Architecture

High-level flow:

  1. Source is added to a workspace.
  2. Content is normalized and chunked.
  3. Chunks are embedded and persisted.
  4. Extraction generates typed graph nodes/edges and workspace summary.
  5. Workspace graph is rendered in 3D/2D UI.
  6. Ask pipeline retrieves relevant chunks + graph neighborhood and returns grounded answer + citations.

Processing can run inline (synchronous dev fallback) or via a dedicated background worker.

Project Structure

Getting Started

1. Install dependencies

npm install

2. Configure environment

Copy .env.example to .env and fill required values:

cp .env.example .env

3. Apply database migrations

Run the SQL migrations in supabase/migrations against your Supabase Postgres instance.

You can do this with Supabase CLI or by executing migration files in order in SQL editor.

4. Create storage bucket

Create a Supabase Storage bucket named sources.

Migrations do not create storage buckets. PDF/GitHub source ingestion expects this bucket to exist.

5. Start development server

npm run dev

Open http://localhost:3000.

Environment Variables

Defined in .env.example:

  • Required:
    • NEXT_PUBLIC_SUPABASE_URL
    • NEXT_PUBLIC_SUPABASE_ANON_KEY
    • SUPABASE_SERVICE_ROLE_KEY
    • GEMINI_API_KEY (required for AI processing/ask)
  • Optional:
    • ELEVENLABS_API_KEY (TTS routes/features)
    • E2E_USER_EMAIL, E2E_USER_PASSWORD, E2E_GITHUB_URL (Playwright scenarios)

Additional optional variables (not currently listed in .env.example):

  • WORKER_MODE (inline default, non-inline for background worker mode)
  • DATABASE_URL (required for pg-boss background worker)

Database and Migrations

Primary schema and API contracts are documented in SCHEMAS.md.

Key schema areas:

  • Core entities: workspaces, sources, source_chunks
  • Graph entities: graph_nodes, graph_edges
  • Conversation/ask: conversations, messages
  • Orchestration: ingestion_jobs
  • Enum contracts and response shapes are treated as locked/stable in SCHEMAS.md

Migration files are in supabase/migrations, including:

  • Initial enums/tables/RLS
  • Chunk upsert + graph metadata
  • Job progress + worker orchestration
  • Conversation types
  • Similarity matching function (match_chunks)

Running the App

Development:

npm run dev

Production build locally:

npm run build
npm run start

Lint:

npm run lint

Background Worker Modes

RC.JS supports two processing modes:

  • Inline mode (default): route handlers execute processing synchronously after enqueue.
  • Background mode: pg-boss queue + worker process handles jobs.

Run worker:

npm run worker

When using background mode, ensure WORKER_MODE is set to a non-inline value and DATABASE_URL is configured.

Seeding Demo Data

Seed a demo workspace:

npm run seed

Seed public-gallery dataset (if needed for demos):

npm run seed:public

Seed scripts are in scripts/seed.ts and scripts/seed-public.ts.

Testing

Unit/integration tests:

npm test

Watch mode:

npm run test:watch

Coverage:

npm run test:coverage

E2E (Playwright):

npm run test:e2e

E2E UI mode:

npm run test:e2e:ui

E2E flows rely on environment values described in e2e/workspace-flow.spec.ts.

API Surface

Major route groups under src/app/api (non-exhaustive):

  • Workspaces:
    • POST /api/workspaces
    • GET /api/workspaces
    • GET /api/workspaces/:id
    • PATCH /api/workspaces/:id
  • Sources:
    • POST /api/workspaces/:id/sources/upload
    • POST /api/workspaces/:id/sources/note
    • POST /api/workspaces/:id/sources/github
    • POST /api/workspaces/:id/sources/slack-import
    • GET /api/workspaces/:id/sources
    • POST /api/workspaces/:id/process
    • GET /api/workspaces/:id/ingestion-status
  • Graph:
    • GET /api/workspaces/:id/graph
    • GET /api/workspaces/:id/graph/node/:nodeId
    • Graph edit/merge endpoints for interactive graph tooling
    • POST /api/workspaces/:id/graph-edit
    • POST /api/workspaces/:id/graph/edge
    • POST /api/workspaces/:id/graph/merge-nodes
  • Ask and conversation:
    • POST /api/workspaces/:id/ask
    • GET /api/workspaces/:id/conversation
    • POST /api/workspaces/:id/builder-chat
  • Public:
    • GET /api/public-workspaces
    • GET /api/public-workspaces/:id
    • POST /api/public-workspaces/:id/fork
    • POST /api/public-workspaces/:id/ask
  • Utility:
    • POST /api/tts

Private endpoints require Authorization: Bearer <supabase_access_token>.

Data Contracts and Schemas

Use SCHEMAS.md as the living source of truth for:

  • Enum and table contracts
  • API response shapes (GraphResponse, AskResponse, SourceListResponse)
  • Cross-team function contracts between backend and AI layers
  • Malleability status (locked/stable/flexible)

Additional implementation and handoff notes are captured in NOTES.md.

Troubleshooting

  • Empty or failing private API calls:
    • Verify bearer token is passed from logged-in session.
    • Verify Supabase URL/keys in .env.
  • Sources stuck in processing:
    • Check WORKER_MODE and whether worker is running when in background mode.
    • Inspect ingestion_jobs and source statuses.
  • Ask returns insufficient evidence:
    • Ensure source chunks were created and embeddings generated.
    • Check Gemini API key and retrieval pipeline logs.
  • Public gallery empty:
    • Seed demo/public data and confirm workspaces are visibility = 'public'.

Roadmap and Known Gaps

From the product and backend specs, remaining improvements include:

  • Complete all UX wiring for create/publish/build actions in every shell path.
  • Further polish demo flow and screenshot/story assets.
  • Harden production deployment and observability setup.
  • Expand connector depth and reliability.

For detailed acceptance checklists and ownership boundaries, see:

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages