Stream list reads the root's reduced state: one getState instead of a DO walk#1449
Merged
Conversation
Every stream/created is already announced to every ancestor stream, but the reducer truncated the announced childPath to the immediate-child segment, so each stream only knew its direct children. Now each stream's reduced core state also keeps the FULL announced path (deduped, insertion order) when it is strictly under the stream's own path. The root stream's descendantPaths therefore becomes the namespace's stream catalog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Core reduced state is persisted in KV and only incrementally caught up, so a reducer that derives NEW fields from already-reduced events (like descendantPaths) would silently leave existing streams without them forever. Introduce CORE_STATE_VERSION, stored next to the state: on wake, a mismatched (or missing) version discards the persisted state and rebuilds it by replaying the full event log from the DO's own SQLite — the same path already used when KV state is lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StreamsCapability.list() walked the namespace stream tree with one
sequential Durable Object call per stream (~4s cold for a 10-stream
namespace on prod, which is what the dashboard streams page waits on).
The root stream's descendantPaths now carries every stream path in the
namespace, so list() is a single getState("/"). descendantPaths is also
exposed on the public StreamState (orpc contract output, legacy adapter,
test stream DO).
The itx worker harness gains real-DO coverage: list() enumerates nested
streams from the root state, and a root persisted before descendantPaths
existed is rebuilt by the CORE_STATE_VERSION replay on its next wake
(verified red without the mechanism).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jonastemplestein
added a commit
that referenced
this pull request
Jun 10, 2026
…etchItxQuery (#1457) ## Problem Converted dashboard routes (streams index) fetch only via `useItxQuery` over the browser's itx WebSocket — route loaders lost their `ensureQueryData` prefetch, so every visit painted a first-visit spinner. ## What this adds (itx DECISIONS **D18**) - **`src/itx/access.ts`** — `accessForPrincipal` / `resolveAccessibleContextId` / `requireWorkerExports` extracted from `fetch.ts` (no behavior change; `resolveAccessibleContextId` now takes `db` instead of the whole `RequestContext`). The `/api/itx` door and the new SSR door share one auth boundary, so they cannot drift. - **`src/itx/server.ts` — `getServerItx(projectSlugOrId, requestContext?)`**: an in-process project-narrowed `Itx` handle built from the request context (`principal` from the auth request middleware, `env` from `cloudflare:workers`, `workerExports` from the context), mirroring exactly what `fetch.ts` does at WebSocket connect time minus the transport. Throws with the kernel's no-existence-probing "not found" wording when the principal may not hold the project. - **`src/itx/loader.ts` — `getLoaderItx`**: isomorphic accessor in the `orpc/client.ts` shape. Server branch dynamically imports `server.ts` (so `cloudflare:workers`/db never enter the browser graph even before the Start compiler strips the branch); browser branch reuses the per-tab socket singleton, moved to `src/itx/react/browser-client.ts` so loaders share the hooks' socket and project-handle cache without importing React. Explicit return types, non-route module — no routeTree Register cycles. - **`prefetchItxQuery`** (same module): best-effort `ensureQueryData` that catches and discards every failure. Prefetch is an optimization, never a gate — this is the guard against the prod incident where a FORBIDDEN thrown during route loading crashed the streams page into the generic error boundary. The component's `useItxQuery` re-surfaces the same error inline. - **`lib/itx-queries.ts`** restructured: each piece of data has ONE `ItxQueryDefinition` (`{project, queryKey, queryFn, staleTime}`) consumed by both the React hook and loader prefetches. - **Exactly one route wired**: the streams index loader seeds the **root stream state**. Breadcrumbs stay lazy by deliberate choice; they're seeded for free via the shared per-path cache keys. ### Note: adaptation to main's drift The original plan said to prefetch `useProjectStreamsList`. Since then the streams index was rewritten around `StreamTreeBrowser` (#1443/#1449), whose first paint gates on the **root stream state**, not the list — and its inline source used ad-hoc keys (`["project", id, "streams", ...]`) that didn't share cache with the breadcrumbs' `itxKey`-based entries. So the loader prefetches `projectStreamStateQuery(root)`, and the route's tree source now keys on `itxKey.project(id, "streams")` — tree nodes, breadcrumb navigators, and the prefetch all land on `projectStreamStateKey(...)` per path. `useProjectStreamsList` keeps its single shared definition. ### DECISIONS numbering The new entry is **D18** (next free number at time of writing). If another in-flight PR also claims D18, renumber on merge. ## SSR seeding verification `router.tsx` wires `setupRouterSsrQueryIntegration({ router, queryClient, wrapQueryClient: true })`, which dehydrates every query seeded during SSR loaders and hydrates it client-side — the same machinery the established `ensureProjectBySlug` beforeLoad pattern already relies on. Nothing there needed changing; the loader awaits `prefetchItxQuery`, so the entry is in the cache before dehydration. ## Tests - `src/itx/loader.test.ts` — prefetch seeds the cache through the resolved handle; swallows handle-resolution failures (no principal / forbidden, leaving component-visible error state in cache); swallows kernel queryFn failures; `getLoaderItx` server-branch wiring. - **New worker harness** `src/durable-objects/itx-server-handle.*` (`pnpm test:itx-server-handle`, same family/convention as `itx-stream-subscribe.*`): hand-built `RequestContext` (principal + real D1 projects row + `ctx.exports`) → `getServerItx` → `itx.streams.list()`/`append()` against a **real Stream Durable Object** through the StreamsCapability loopback. Covers member by slug + id, admin, stranger (denied), anonymous (denied), unknown slug (denied) — the insufficient-principal SSR path is exercised here and in the unit suite. - `pnpm vitest run src/itx/react` — 22 tests green (plus loader tests). - Repo root: `pnpm typecheck && pnpm lint && pnpm format && pnpm test` all green. Docs: D18 in `src/itx/DECISIONS.md`, SSR section of `docs/itx-orpc-replacement-plan.md` marked done, `tasks/os-orpc-teardown.md` checkbox ticked. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches authentication boundaries for SSR and /api/itx (shared module) and changes loader error handling; well-covered by new harness and loader tests, but auth drift or silent prefetch failures could affect first paint or error UX. > > **Overview** > Adds **SSR/loader prefetch** for itx so TanStack routes can seed React Query without opening a WebSocket, restoring the prefetch behavior lost when streams moved off oRPC. > > **Shared auth boundary** — `access.ts` centralizes `accessForPrincipal`, `resolveAccessibleContextId`, and `requireWorkerExports`; `/api/itx` in `fetch.ts` and the new SSR path both use it so connect-time and loader-time access cannot drift. > > **In-process SSR handle** — `getServerItx` builds a project-narrowed `Itx` from request context via the same resolve chain as WebSocket connect (no capnweb). **`getLoaderItx`** is the isomorphic entry (server → dynamic `getServerItx`; client → shared tab singleton in `browser-client.ts`). **`prefetchItxQuery`** runs `ensureQueryData` but **swallows all errors** so forbidden/missing projects do not blow up route loaders (prod streams error-boundary fix). > > **Query definitions** — `ItxQueryDefinition` in `lib/itx-queries.ts` is shared by hooks and loaders; the **streams index** loader prefetches root stream state and aligns tree `itxKey` usage with breadcrumbs. > > **Tests** — Unit tests for loader prefetch contract; new `test:itx-server-handle` worker harness proving `getServerItx` → real Stream DO + auth denial cases. Minor e2e timeout/assertion tweaks. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4a96cca. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- CLOUDFLARE_PREVIEW --> ## Environment Config Lease <!-- CLOUDFLARE_PREVIEW_STATE --> <!-- { "apps": { "os": { "appDisplayName": "OS", "appSlug": "os", "status": "deployed", "updatedAt": "2026-06-10T17:57:50.268Z", "headSha": "4a96cca6c8bd77ec2ad5dc15db252310591bf1d8", "message": null, "publicUrl": "https://os.iterate-preview-4.com", "runUrl": "https://github.com/iterate/iterate/actions/runs/27295319125", "shortSha": "4a96cca" }, "semaphore": { "appDisplayName": "Semaphore", "appSlug": "semaphore", "status": "deployed", "updatedAt": "2026-06-10T17:56:32.669Z", "headSha": "4a96cca6c8bd77ec2ad5dc15db252310591bf1d8", "message": null, "publicUrl": "https://semaphore.iterate-preview-4.com", "runUrl": "https://github.com/iterate/iterate/actions/runs/27295319125", "shortSha": "4a96cca" } }, "environmentConfigLease": { "dopplerConfig": "preview_4", "leasedUntil": 1781117677160, "leaseId": "a51db8c6-cf17-4ffe-b115-af704299778a", "slug": "preview-4", "type": "environment-config-lease" } } --> <!-- /CLOUDFLARE_PREVIEW_STATE --> Lease: `preview-4` Doppler config: `preview_4` Type: `environment-config-lease` Leased until: 2026-06-10T18:54:37.160Z ### OS Status: deployed Commit: `4a96cca` Preview: https://os.iterate-preview-4.com [Workflow run](https://github.com/iterate/iterate/actions/runs/27295319125) Updated: 2026-06-10T17:57:50.268Z ### Semaphore Status: deployed Commit: `4a96cca` Preview: https://semaphore.iterate-preview-4.com [Workflow run](https://github.com/iterate/iterate/actions/runs/27295319125) Updated: 2026-06-10T17:56:32.669Z <!-- /CLOUDFLARE_PREVIEW --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
StreamsCapability.list()enumerated a namespace by recursively walkinggetState("/")→ child → child, one sequential Durable Object call per stream. Measured on prod: ~3,985 ms cold for a 10-stream namespace (each cold DO wake ~400 ms, serialized). This backs the dashboard streams page, so the first view of a project took seconds.Change
Every
stream/createdis already announced to every ancestor stream (including/) viachild-stream-created; the reducer just threw the full path away after truncating it to the immediate-child segment.packages/streams/src/processors/core/{contract,implementation}.ts): each stream's reduced state now also accumulatesdescendantPaths— the full announced paths strictly under its own path, deduped, insertion order. The root's copy is the namespace's stream catalog.apps/os/src/domains/streams/entrypoints/streams-capability.ts):list()is now a singlegetState("/")→["/", ...descendantPaths]. The recursivelistNamespaceStreamPathswalk is deleted.listChildrenstill useschildPaths, unchanged.descendantPathsadded to the sharedStreamStatezod (orpcstreams.createoutput), the legacy adapter (toLegacyStreamState), and the test stream DO.Migration
Core reduced state is persisted in KV and only incrementally caught up, so already-reduced events would never re-reduce and existing streams would be missing
descendantPathsforever. There was no existing version/replay mechanism for the DO's persisted core state (the browserschemaVersionconstants are for browser-side SQLite caches only), so this adds a minimal one:CORE_STATE_VERSION(now2) stored in KV next to the state.stateVersionkey), so every existing stream re-reduces on its next wake. No fallback walk inlist(); the root state is authoritative.Tests
packages/streamsreducer coverage: root and intermediate ancestors accumulate full descendant paths; dedupe; self/sibling/shared-prefix announcements ignored; replay/catch-up tests extended.pnpm test:itx-stream-subscribe):list()returns/, intermediate and nested streams after creating…/a,…/a/b,…/c; and a root whose persisted state was rewritten to the pre-descendantPathsshape (viarunInDurableObject) is rebuilt by the version replay on its next wake — this test was verified to go red with the version mechanism disabled.packages/streamstests (59),packages/sharedtests,apps/ospnpm test(226),test:itx-stream-subscribe(8),test:project-ingress(6),test:codemode-session(18),test:project-mcp-server-connection(4),pnpm typecheck,pnpm lint,pnpm format.🤖 Generated with Claude Code
Note
Medium Risk
Touches core stream reduced state, DO wake/replay, and a hot dashboard path; migration relies on event-log replay on next wake rather than a list-time fallback walk.
Overview
Namespace stream listing is now one root Durable Object read instead of a recursive walk that called
getStateon every stream in sequence (a major latency win for dashboards).The core reducer adds
descendantPaths: on eachchild-stream-createdannouncement it keeps the full announced path (deduped), whilechildPathsstays for immediate children (listChildrenunchanged). The root stream’sdescendantPathsis the namespace catalog;StreamsCapability.list()returns["/", ...rootState.descendantPaths]and removeslistNamespaceStreamPaths.Migration: the Stream DO persists
CORE_STATE_VERSION(2) beside KV state. On wake, a missing or stale version replays the SQLite event log to rebuild reduced state (includingdescendantPaths) instead of trusting old KV.descendantPathsis also exposed on sharedStreamState, the legacy adapter, and tests (including real-DO list + replay-after-downgrade scenarios).Reviewed by Cursor Bugbot for commit bb76744. Bugbot is set up for automated code reviews on this repo. Configure here.
Environment Config Lease
No active environment config lease.
OS
Status: released
Commit:
bb76744Preview: https://os.iterate-preview-2.com
Summary: Preview app released.
Workflow run
Updated: 2026-06-10T13:38:20.608Z
Semaphore
Status: released
Commit:
bb76744Preview: https://semaphore.iterate-preview-2.com
Summary: Preview app released.
Workflow run
Updated: 2026-06-10T13:38:12.157Z