Class-based stream processors: crisp batch model, honest contracts, regression tests by jonastemplestein · Pull Request #1401 · iterate/iterate

jonastemplestein · 2026-06-09T12:16:25Z

Summary

Migrates the first stream processors onto a class-based StreamProcessor model (replacing the earlier prototype shape) and tightens it into a design that's explainable in one sentence:

The host feeds ordered event batches into ingest, the base class reduces each new event into state, hands the batch to the process* hooks for side effects, and checkpoints state + offset once all blocking work completes.

The model

@iterate-com/streams/stream-processor exports the abstract StreamProcessor<Contract, Deps>. Contracts are data (schemas, event catalogs, consumes/emits); processor classes own reduction and side effects; hosts own transport and checkpoint storage (readState/writeState, with an in-memory default for tests).

The naming draws one deliberate line:

ingest — the host-facing sink. Serializes batches (a later batch never starts until the previous one completed or failed) and must not be overridden, because the serialization guarantee lives there. Enforced by lint.
The process* hook family — the authoring surface, all running inside the serialized section:
- reduce — pure projection of one consumed event into the next state
- processEvent — synchronous per-event side effects; what most processors implement
- processEventBatch — batch-level side effects (e.g. one SQLite transaction over the already-deduped new events); default implementation calls processEvent per reduced event
- prepare — optional one-time setup that must land before the checkpoint is first read (e.g. schema migrations that reset it)

Async side effects are explicit: blockProcessorWhile holds the checkpoint (and the next batch), runInBackground is fire-and-forget with logged failures. The checkpoint is written only after hooks and blocking work succeed — at-least-once redelivery on failure.

Contracts and type inference

Reducers moved off contracts onto classes; contract values/types are capitalized (CoreProcessorContract).
Hook overrides annotate args as Parameters<StreamProcessor<Contract>["method"]>[0] — the single sanctioned spelling, enforced by a new iterate/stream-processor-override-args lint rule (which also catches near-miss hook names and ingest overrides). The arg shapes are deliberately not exported.
Hooks receive plain ProcessorState/ConsumedEvent types (no DeepReadonly — it forced a cast in every real subclass) and must treat them as immutable.
Honest wildcard types: consumes: ["*", ...named] previously inferred the named-only union, so the switch fallthrough was typed never while reachable at runtime. The union now includes WildcardConsumedEvent (type-level literal "*"): named events keep exact payload narrowing, the wildcard branch is reachable with an unknown payload. ["*"] alone stays plain StreamEvent; named-only contracts stay exhaustive; "*" in emits is rejected at the definition site. This lines up with planned consumes-driven subscription filtering.

Hosts

Browser: views register slug + createProcessor; checkpoints live in a shared processor_state table (browserProcessorStateStorage), separate from each processor's projection tables. BrowserRawEventsProcessor and BrowserEventFeedProcessor batch their SQLite writes in processEventBatch overrides. A schema-version bump clears the checkpoint together with the dropped projection table via prepare() — before the resume cursor is decided.
Stream DO: CoreStreamProcessor replaces the old object-literal built-in. The DO runs it inline during append via validateAppend / reduceEvent / processReducedEvent — an explicitly core-only surface built on the protected reduceRawEvent helper, keeping the base class a pure batch/checkpoint model.

Tests

stream-processor-class.test.ts — runtime contract: batch serialization, checkpoint semantics (one write per batch, none on failure, at-least-once redelivery), offset dedupe, blocking/background side-effect behavior, blocking-work settlement on hook throw, readState failure retry, wildcard runtime semantics.
browser/processor-state-storage.test.ts — real SQLite via node:sqlite: checkpoint round-trip and delete semantics, the schema-version-bump reset (including direct-ingest-without-snapshot), replay dedupe through the append-continuity trigger.
stream-processor-types.test.ts — compile-time: dependency-event inference into all hooks, wildcard-only/mixed/named-only consume typing, consumes-typo and wildcard-emits rejection.

Design notes: apps/os/tasks/stream-processor-class-design-notes.md.

Validation

pnpm --dir packages/streams typecheck && pnpm --dir packages/streams test (63 tests)
pnpm --dir packages/streams/example-app typecheck
pnpm --dir apps/os typecheck
pnpm lint && pnpm format

🤖 Generated with Claude Code

Environment Config Lease

No active environment config lease.

OS

Status: released
Commit: 41762ad
Preview: https://os.iterate-preview-4.com
Summary: Preview app released.
Workflow run
Updated: 2026-06-09T22:01:55.377Z

Note

High Risk
Changes append validation and reduction on the Stream Durable Object and browser mirror resume/checkpoint behavior; incorrect checkpoint or schema-reset logic could skip replay or wedge local SQLite mirrors.

Overview
Introduces a class-based StreamProcessor API in @iterate-com/streams and migrates the first hosts off the old implementProcessor + runner shape.

Base model: hosts call ingest (serialized batches, non-overridable); subclasses implement reduce, processEvent, and/or processEventBatch, with optional prepare before the checkpoint is read. Checkpoints advance only after hooks and blockProcessorWhile work succeed; runInBackground is best-effort. Reducers move off contracts onto processor classes; contracts are renamed to capitalized exports (e.g. CoreProcessorContract). A new iterate/stream-processor-override-args lint rule enforces hook arg typing and blocks overriding ingest.

Browser: views pass slug + createProcessor instead of a processor object and loadCheckpoint. Checkpoints use a shared processor_state table via browserProcessorStateStorage; BrowserRawEventsProcessor / BrowserEventFeedProcessor batch SQLite writes in processEventBatch. Schema version bumps clear processor_state with the projection table so stale offsets cannot skip replay.

Stream DO: CoreStreamProcessor replaces the inline built-in processor; append uses validateAppend, reduceEvent, and processReducedEvent (child-stream propagation via runInBackground on stream/created).

Types: mixed consumes: ["*", ...named] now includes WildcardConsumedEvent so wildcard switch branches are reachable. Added runtime/type tests and design notes under apps/os/tasks/stream-processor-class-design-notes.md.

^{Reviewed by Cursor Bugbot for commit dc57d28. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2116d85. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9b8460d. Configure here.}

# Conflicts: # packages/streams/example-app/src/routes/-event-feed-view.tsx # packages/streams/package.json

…ghten types - Fix latent browser mirror bug: the raw-events resume checkpoint now lives in processor_state, so the PRAGMA user_version reset must clear it alongside the dropped events table — otherwise a future schema bump leaves a stale cursor over an empty table, skips historical replay, and wedges on the continuity trigger. The processor also ensures its schema before the first snapshot read so the reset happens before the replay cursor is reported. - deleteBrowserProcessorState with no subscriptionKey now clears all rows for the slug. - Inline StreamProcessor#reduce: drop the fake-contract round-trip through runProcessorReduce in favor of direct consumed-event lookup + payload parse + this.reduce + object-state assert. - Make Parameters<StreamProcessor<C>["method"]>[0] the only override-arg spelling: ReduceArgs/ProcessEventArgs/ReducedEvent are no longer exported. - Drop the unused _Self generic on StreamProcessorContract and the pointless <Result> generics on void-returning work runners. - Settle already-registered blocking work when processEvent throws mid-batch so rejections can't go unobserved; retry #loadState after a failed snapshot read instead of caching the rejection forever; make per-batch blocking work a local instead of instance state. - CoreStreamProcessor.reduce: replace the entry stateSchema.parse typecast with a plain cast (one parse per event instead of two on the append hot path) and drop the dead `next ?? state`. - Refresh stale design notes (processEvents is gone; inline methods are reduceEvent/processReducedEvent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…nly, core-owned inline surface The base class is now explainable in one sentence: the host pushes ordered batches into processEventBatch, the base reduces each new event into state, hands the batch to processBatch for side effects, and checkpoints once all blocking work completes. - New protected async processBatch hook, called inside the serialized section after the whole batch is reduced and before the checkpoint is written. The default implementation calls processEvent per reduced event. Overriding processEventBatch is now a lint error (like processEvents): previously the prescribed override-then-super pattern ran subclass SQL writes outside the serialization queue, racing concurrent batch deliveries from the DO's fire-and-forget pump and only surviving via idempotent writes. - Browser processors override processBatch and lose the snapshot-then-filter dance: args.events is already deduped past the checkpoint, and args.previousState replaces the checkpoint state read. - Remove DeepReadonly. It forced an escape-hatch cast in every real subclass (feed: `as FeedState`, core: parse-as-typecast). Hooks receive plain ProcessorState/ConsumedEvent and must treat them as immutable. - Move the inline DO surface (reduceEvent, processReducedEvent) off the base class onto CoreStreamProcessor, its only user, built on the new protected reduceRawEvent helper. The base class is a pure batch/checkpoint model. - Drop the unused processorKey constructor arg and getter. - processBatch errors now also settle already-registered blocking work before the batch rejects. - Update the override-args lint rule, type tests, and design notes for the new shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…types Naming: the host-facing serialized sink is now `ingest` — deliberately outside the hook family — and the author-facing hooks are a consistent `process*` family: `processEvent` (per event, what most processors implement) and `processEventBatch` (batch-level, default loops processEvent). No more processEventBatch-vs-processBatch ambiguity. The lint rule forbids defining ingest/processEvents/processBatch on subclasses and type-checks the three real hooks. Wildcard consume typing: `consumes: ["*", ...named]` previously inferred the named-only union, so the fallthrough branch was typed `never` while being reachable at runtime — an assertNever there would have thrown on every unnamed event. The union now includes WildcardConsumedEvent, whose `type` is the literal "*": named events keep exact payload narrowing (TS only filters union members on non-matching unit discriminants, verified empirically), and the wildcard branch is reachable with an unknown payload. `["*"]` alone stays plain StreamEvent; named-only contracts stay exhaustive. This also lines up with the planned consumes-driven subscription filtering, where "*" is the explicit opt-in for unfiltered delivery. Type tests now cover: dependency-event inference, wildcard-only, mixed wildcard narrowing (named payloads exact, default branch never-free), named-only exhaustiveness, consumes typo rejection, and "*"-in-emits rejection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two new suites pinning the behaviors the next migrations will lean on: stream-processor-class.test.ts — the StreamProcessor base contract: - reduce + checkpoint: one writeState per batch, schema-parsed resume from readState, offset dedupe within and across batches, no checkpoint write for fully-replayed batches, non-consumed events advance the checkpoint without running hooks - hook wiring: default processEventBatch fans out to processEvent with correct per-event previousState/state chains; processEventBatch overrides receive pre-deduped events plus batch-entry/exit state - serialization: a later ingest never starts until the previous batch settles; a failed batch is not checkpointed, does not poison the queue, and redelivers at-least-once - side effects: blockProcessorWhile holds the checkpoint and fails the batch on rejection; blocking work registered before a hook throw is settled before the batch rejects; runInBackground logs failures without holding the checkpoint; keepAliveWhile wraps both - storage: failed readState rejects the batch and retries on the next ingest; in-memory fallback round-trips snapshots - wildcard runtime: named payloads are schema-validated (bad payload fails the batch), unnamed events flow through the "*" branch browser/processor-state-storage.test.ts — real SQLite via node:sqlite: - processor_state round-trip keyed by slug + subscription key; single-key vs all-keys delete - the schema-version-bump regression: a fresh client over an old-version database must clear the checkpoint with the dropped events table, report offset 0, and rebuild the mirror from full replay (previously: stale checkpoint over an empty mirror, wedged on the continuity trigger) - same-version reload resumes from the checkpoint and dedupes replayed offsets - the append-continuity trigger rejects offset gaps Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…comments - Document every exported type and accessor on the StreamProcessor surface: iterate context, contract slice, base deps, snapshot/storage shapes, constructor args, state/checkpointOffset/snapshot, and why #runKeepAliveBackedWork bridges fire-and-forget keep-alives into promises. - Document the newly-exported shared helpers (assertObjectProcessorState, getConsumedEventDefinition) — the runtime counterpart of ConsumedEvent. - CoreStreamProcessor: rewrite the broken validateAppend comment block into a docstring (pause door semantics, why policy lives in other processors, future permissions); per-method docs on the inline reduceEvent / processReducedEvent surface; note that the exit stateSchema.parse validates every transition; replace the single-case switch in processEvent with an early return and explain the ancestor child-stream propagation. - Class docstrings on BrowserRawEventsProcessor and BrowserEventFeedProcessor (including how reduce and processEventBatch stay in lockstep via planFeedOps). - Module note on processor-state-storage: why checkpoints live apart from projection tables and the at-least-once consequence; docstrings on browserProcessorStateStorage and BrowserHostedProcessor. - Lint rule: document the deliberate text-based contract-type extraction and its limits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…s before the first read Bugbot caught a real gap: the raw-events schema reset ran via a snapshot() override and inside processEventBatch, both of which can land after #loadState has already memoized the checkpoint — a host calling ingest without a prior snapshot() would resume from a stale cursor over a freshly-reset table and trip the continuity trigger. The base class now exposes a one-time protected prepare() hook, awaited inside #loadState before readState, so setup that can invalidate the checkpoint always lands before the resume cursor is decided. Both browser processors move their schema ensure into prepare(); the raw-events snapshot() override and the redundant in-batch ensure calls are gone. prepare() failures reject the triggering call and retry on the next one (same path as readState failures). New regression test covers the exact reported path: a version bump followed by a direct ingest (no snapshot) must reset, replay from offset 1, and rebuild the mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ile instance The localMaxOffset < 0 early-return was a relic of the old loadCheckpoint API that returned -1 for an empty mirror; class snapshots start at 0, so the guard never fired and fresh mirrors paid a pointless runtimeState RPC. Now <= 0. Also document why reconcile creates a throwaway processor instance: checkpoints memoize on first read, so the real instance must be born after any discard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…y messages "ingest/processEvents/processBatch" read as three mysterious banned names. They are two different things: ingest is the host-facing serialized sink that must stay on the base class, and processEvents/processBatch are near-miss hook names that do not exist (defining them would silently never be called). Each case now gets its own message naming the real hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fixes both outstanding Bugbot findings on the browser hosting path: Atomic checkpoints (mirror reconcile misses stale offsets): projection writes and the processor_state checkpoint were two separate statements, so a crash between them left the mirror ahead of the stored cursor — reconcile and replayAfterOffset decisions made from snapshot() could then miss a server rewind or trip the continuity trigger. processor-state-storage now exports upsertProcessorStateStatement, and both browser processors include it in their projection transaction, so the checkpoint can never lag the mirror. The base class's later writeState persists the same snapshot again (idempotent upsert). Processor deps gain the subscriptionKey so the upsert targets the same row as the host's storage; views pass it through. Leader shutdown (teardown skips processor shutdown): StreamProcessor gains shutdown(), which stops accepting batches and resolves once the in-flight batch settles. stopSubscriptionElection now releases the writer lock only after shutdown completes — leadership is the lock, so a successor can never overlap with the old processor's writes. Regression tests: simulated crash between the projection transaction and writeState (checkpoint must already be durable, fresh load resumes at the mirror's head), and shutdown semantics (waits for in-flight work, rejects further ingests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…own" This reverts commit 41762ad.

…scriptions, legacy model deleted (#1402) ## Summary Complete rewrite of the original spike: every stream processor apps/os hosts now runs on the class-based `StreamProcessor` model (#1401), owned directly by its domain Durable Object, and the Stream DO reaches subscribers through the `packages/shared` Callable abstraction. All legacy processor-model code is deleted (−17k lines net). Rebased onto main including the itx capabilities work (#1407). Full design rationale and issue log: `apps/os/tasks/stream-processor-class-migration-log.md` (D1–D12, I1–I6 + deletion inventory), plus per-domain notes under `apps/os/tasks/migration-notes/`. ### Hosting model A Durable Object hosts processors as plain class fields: ```ts export class AgentDurableObject extends DurableObject<Env> { host = createStreamProcessorHost(this.ctx); agent = this.host.add("agent", (deps) => new AgentProcessor({ ...deps, ... })); chat = this.host.add("agent-chat", (deps) => new AgentChatProcessor(deps)); async requestStreamSubscription(args: RequestStreamSubscriptionArgs) { await this.ensureStartedOrInitializeFromRuntimeName(); return await this.host.requestStreamSubscription(args); } } ``` `createStreamProcessorHost` provides checkpoint storage in DO KV (keyed by processor name), a late-bound stream context, per-subscription side-effect anchoring, and `processor-registered` announcements. Any number of named processors per DO. ### Subscriber delivery via Callable `stream/subscription-configured` payloads carry `{ type: "callable", callable: Callable }`; the Stream DO dispatches the callable with the subscription handshake (`dispatchCallable` — same Workers RPC transport, so the live stream stub passes through). The hardcoded `STREAM_PROCESSOR_RUNNER` dialing is gone; `packages/streams` now depends on `packages/shared`. Legacy `built-in` subscriber shapes reduce harmlessly but are no longer dialed; OS re-appends callable subscriptions through the existing ensure-on-access paths with new idempotency keys. No subscriber authorization yet (explicitly out of scope). ### Contract-driven event filtering `subscribe`/`subscribeOutbound` accept `eventTypes`; the pump filters post-read while its cursor advances past non-matching events. Hosts always pass `contract.consumes` — the contract is the filter (`"*"` = unfiltered). Catch-up helpers wait on consumed-event targets instead of the raw stream head. ### Side-effect anchor `StreamProcessor` gains `sideEffectsAfterOffset`: events at or below the anchor (persisted at first subscription handshake) reduce into state but skip side effects — attaching to an existing stream rebuilds state without re-firing historical effects (e.g. old LLM requests). Replaces the legacy dual-cursor + first-attach lookback machinery. ### Migrated processors (all wire-format-identical) | Host DO | Processors | |---|---| | `AgentDurableObject` | agent, agent-chat, openai-ws, cloudflare-ai, jsonata-reactor, agent-host | | `ProjectDurableObject` | project-lifecycle | | `RepoDurableObject` | repo | | `SlackIntegrationDurableObject` / `SlackAgentDurableObject` | slack / slack-agent | | `CodemodeSession` | codemode | | streams staging runner | echo-example, circuit-breaker | openai-ws keeps its socket as processor instance state (the DO is the connection scope). `scheduling`, `jsonata-transformer`, `dynamic-worker` had no live subscription path and were deleted, not migrated. ### LLM requests are background work (D12) The LLM providers do NOT hold the processor's batch queue while a request is in flight. Executing under `blockProcessorWhile` would mean a cancellation or superseding event physically cannot be reduced until the request it should affect has finished — defeating the staleness check (`isAgentLlmRequestStillCurrent`) both providers run before appending agent-visible output. Instead: - `agent/llm-request-requested` executes via `runInBackground` (keep-alive-backed through the host's `ctx.waitUntil`), so subsequent events keep flowing while requests run; stale requests complete silently. - Crash recovery no longer rides on checkpoint-held redelivery (the checkpoint now passes the request immediately). Each provider tracks in-flight request ids in instance state; a `started` entry in reduced state with no in-flight execution marks a request a previous incarnation abandoned, and the next delivered batch re-executes it from stream history — still guarded by the staleness check. ### Deleted The legacy OS `StreamProcessorRunner` DO (+ binding; alchemy computes `deleted_classes` automatically), `packages/shared/src/stream-processors/**` (~47 files), the shared DO mixins, and the legacy runner model in `packages/streams` (`processor.ts`, `processor-runner.ts`, `standard-processor-behavior.ts`, ~14 legacy-only helpers). ~35 importers repointed. ### Issues found and fixed during validation & review - **I5** — the agent wake hook awaited processor catch-up inside `blockConcurrencyWhile`; with processors co-hosted on the same DO this deadlocks against the input gate (the handshake/delivery it waits for queues behind it). Rule recorded in the log: never await processor catch-up inside a lifecycle gate on a co-hosting DO. - **I6** — the long-flaky streams-e2e tail spec, root-caused from a CI trace: TanStack Virtual's end target sits ~12px short of the true bottom on Linux font metrics (≤2px on macOS), so the tail-pin's scroll-delta "user left" heuristic fired on the virtualizer's own reconcile writes. The pin now releases only on real user-input signals; CI uploads playwright traces on failure. - Stream child-path announcements and dials to uninitialized codemode sessions on routed streams (found via live-preview e2e). - Review (Bugbot): Cloudflare AI now retries requests a crashed incarnation left in `started` (skip only `completed`), with regression tests; cold `AgentDurableObject` instances initialize their lifecycle before accepting subscription handshakes; `agent-host` wakes its agent once per incarnation on any delivered event instead of an anchor-skipped historical `stream/created`. ## Validation - `apps/os`: typecheck 0 errors, 167 unit tests; workerd suites `test:project-ingress` 6/6 and `test:codemode-session` 18/18 (cover the callable handshake, routed streams, and itx capabilities end-to-end) - `packages/streams` (58), `packages/shared` (64), `packages/ui`, example-app: typecheck + tests green - `pnpm lint` / `pnpm format` clean - **Live preview acceptance** (`os.iterate-preview-4.com`): full OS e2e suite against the deployment — 27 passed / 1 todo / 0 failures, including real OpenAI conversations on freshly created projects, Cloudflare AI Gateway, codemode script execution, and routed Slack webhook → bang-command replies against the real Slack API 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Environment Config Lease    Lease: `preview-2` Doppler config: `preview_2` Type: `environment-config-lease` Leased until: 2026-06-10T06:56:41.972Z ### OS Status: awaiting tests Commit: `4d30dbd` Preview: https://os.iterate-preview-2.com [Workflow run](https://github.com/iterate/iterate/actions/runs/27256373612) Updated: 2026-06-10T05:56:44.340Z ### Semaphore Status: awaiting tests Commit: `4d30dbd` Preview: https://semaphore.iterate-preview-2.com [Workflow run](https://github.com/iterate/iterate/actions/runs/27256373612) Updated: 2026-06-10T05:56:44.300Z  --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…capnweb pointers, fix task states (#1432) Documentation sweep over `apps/os`. Every statement written into a doc was verified against the code on this branch. ## Changes **`apps/os/README.md` (= `AGENTS.md`)** - Important Files: `src/app.ts` / `src/entry.workerd.ts` do not exist — replaced with `src/worker.ts` (Worker entrypoint) and `src/config.ts` (`AppConfig` schema). All other listed files verified to exist. - Real-worker tests: the documented vitest configs (`src/capnweb/e2e/vitest.config.ts`, `src/domains/capability-prototype/e2e.vitest.config.ts`) are gone — replaced with the real lanes `pnpm e2e` (`e2e/vitest.config.ts`) and `pnpm e2e:itx` (`src/itx/e2e/vitest.config.ts`), verified against `apps/os/package.json`. - `pnpm cf:deploy # production deploy` was wrong and dangerous: `cf:deploy` deploys to whatever Doppler/Alchemy stage is ambient. Now documents both `cf:deploy` (ambient stage) and `pnpm deploy` (the `doppler --config prd` wrapper). - Removed the nonexistent `/org/:organizationSlug` route; remaining routes verified against `src/routes/`; added `/new-project`. **`apps/os/CONTEXT.md`** — fixed the example-dialogue claim that organization UI lives under `/org/:organizationSlug` (no such route; orgs live in the auth worker). **`apps/os/docs/architecture-and-operations.md`** — rewritten. The old doc described the pre-migration world: Clerk auth (whole `## Clerk` section, `sync-clerk-apps.ts`, `APP_CONFIG_CLERK__*`), `/orgs/:organizationSlug` route maps, inbound MCP via `ProjectMcpServerEntrypoint` (now a hardcoded 410 tombstone), wrong redirect claims, and an unprefixed `/durable-objects/stream` debug route. The new doc describes current reality: `src/worker.ts` dispatch pipeline, Iterate Auth middleware, real route map and root-redirect behavior (`/` → `/projects/$projectSlug` or `/projects`; project root renders `ProjectHomePage`), canonical MCP endpoint from `APP_CONFIG_MCP__BASE_URL` with Iterate Auth protected-resource metadata, `/__durable-objects/<kind>/<name>/<path>` debug proxy (kinds verified), itx endpoints, `scripts/sync-auth-clients.ts`, current codemode default/example providers, and current smoke-test env vars (verified in the e2e test files). **`apps/os/docs/headless-local-debugging.md`** — `/projects/new` → the real route `/new-project`. **`apps/os/docs/iterate-context.md`, `iterate-context-learnings.md`** — both pointed at the deleted `src/capnweb/` tree as "the current design"; now short tombstones pointing at the successor (`src/itx/` README + DECISIONS, `docs/itx-spec.md`). **`apps/os/docs/capability-system-research-and-design-notes.md`, `rpc-target-constructor-shape-research.md`** — added status headers marking them historical research notes superseded by itx; bodies untouched. **`apps/os/src/itx/README.md` + `src/itx/handle.ts`** — the "Typed caps" `ProjectCaps` declaration-merging pattern does not exist in code (no `ProjectCaps` interface anywhere). Rewrote the README section to the thing that actually works: casting `itx.cap("name")` through the exported `Stubify<T>` type. Also fixed the same false claim in the `Stubify` doc comment in `handle.ts` (comment-only change). **`apps/os/docs/itx-spec.md`** — status header said "IMPLEMENTED on the `itx-implementation` branch"; PR #1407 is merged to main (verified in git history). Marked the one known divergence honestly: the §6.3 client reconnect loop was never built — `connectItx` (`src/itx/client.ts`) is one-shot, and there is no `itx.cap.disconnected` event. Corrected §6.3 and the related §4 caveat. **`apps/os/tasks/`** - Deleted `simplify-context-cloudflare-native.md` (state: todo, but shipped — `src/worker.ts` imports `env` from `cloudflare:workers` directly, `RequestContext` is the narrow request-scoped shape the task specified, auth lives in Start request middleware, the manifest/`src/app.ts` is gone). - Deleted `project-egress-secrets-mvp.md` (state: todo, but shipped — `ProjectEgress` entrypoint, `ProjectDurableObject.egressFetch` with `substituteProjectEgressSecretHeaders`, D1-backed `SecretsCapability.getSecret`, and the `/api/itx/egress-echo` echo proof covered by `src/itx/e2e/itx-egress.e2e.test.ts`). - Grooming rules (`docs/tasks-grooming.md`) say "Delete when done", so deletion rather than state edits. - Added brief status notes (no rewrite) to `codemode-session-vertical-slice.md` (checked-off "tiny worker" box diverged: `CodemodeSession` lives in the main OS worker) and `codemode-session-night-plan.md` (plan superseded by itx). ## Skipped - Nothing skipped; all nine items verified and addressed. ## Flags for reviewers - `src/itx/handle.ts` got a comment-only edit (the `Stubify` doc comment made the same false declaration-merging claim as the README). No runtime change; typecheck/lint/tests pass. - The two deleted task files: please sanity-check the "shipped" verdicts above if you have more context on intended remaining scope. - Carve-outs respected: no changes to the streams type systems or to how the os-streams worker is deployed. ## Checks - `pnpm install`, `pnpm format` (oxfmt), `pnpm typecheck`, `pnpm lint`, `pnpm test` — all pass. ## Task-file audit A follow-up commit deletes 22 task files whose work was verified as shipped, obsolete, or purely historical. (Two more from the audit — `apps/os/tasks/project-egress-secrets-mvp.md` and `apps/os/tasks/simplify-context-cloudflare-native.md` — were already deleted by earlier commits on this branch, see above.) ### Deleted: completed - `tasks/cf-prd-orphaned-resources-cleanup.md` — live Cloudflare API check of the prd account (2026-06-10) shows 14 worker scripts (was 1026 at the task's 2026-05-18 sweep) and 6 D1 databases; cleanup is done. - `tasks/complete/2026-05-22-os-captun-worker-test-tunnel.md` — shipped via merged PR #1361 ("codemode++ e2e++"); all described artifacts exist on main and survived the golden-path rebuild (#1411). - `tasks/dead-code-and-docs-cleanup-audit.md` — high-confidence items all shipped; `pnpm-workspace.yaml` no longer lists the dead packages and now uses `apps/*`/`packages/*` globs. - `tasks/os-auth-spurious-logout-refresh.md` — commit ad6da76 "Fix 5-min logout, deploy-time JWKS, and stream append skeleton flash (#1410)" (merged 2026-06-10) shipped exactly this work. - `tasks/os-codemode-router.md` — task file was added in the very PR that implemented it (commit 98ee148, #1294). - `tasks/os-domain-capability-orpc-refactor-design.md` — every major pillar of the design (domains layout, capabilities, oRPC structure) exists on main. - `tasks/os-domain-capability-orpc-refactor-prd.md` — shipped in PR #1305 "Make codemode function calls event-driven" (squash commit 284193e, merged 2026-05-08). - `tasks/semaphore-lease-renewal.md` — the described lease-renewal feature exists on main as `resources.renew` (named "renew" rather than the proposed "extend") in `apps/semaphore`. - `tasks/signup-slug-uniqueness.md` — shipped with the auth worker (PR #1273); `packages/shared/src/slug.ts` implements `resolveUniqueSlug`/`slugifyWithSuffix`. - `apps/os/tasks/codemode-session-night-plan.md` — planned outcomes verifiably shipped on main, in evolved form (codemode session browser UI and follow-ons). - `apps/os/tasks/codemode-session-vertical-slice.md` — all 11 ticked checklist items shipped via PRs #1294/#1305 and follow-ups. - `apps/os/tasks/refactor-lifecycle-init-params-as-structured-name.md` — every acceptance criterion implemented in the `with-lifecycle-hooks.ts` mixin on main. - `apps/os/tasks/repos-vertical-slice.md` — frontmatter already says `state: done` and the described slice verifiably exists on main. - `apps/os/tasks/slack-processor-unwind.md` — all target-shape items exist on main (`/integrations/slack` stream path; no `/integrations/slack/webhooks` references). ### Deleted: obsolete / nonsense - `tasks/github-oauth-use-repo-id.md` — all referenced code is gone: `linkExternalIdToGroups` / `repoId` / `repository.id` return zero hits repo-wide. - `tasks/ignoreme-email-security.md` — every code path the task targets was deleted with the legacy OS1 stack (commit 545854d, #1341). - `tasks/os-stream-runtime-big-refactors.md` — os2-era brainstorm list largely superseded or done differently; item 2 shipped via PR #1394. - `tasks/realtime-pusher-efficiency.md` — targets the legacy OS1 realtime pusher, which no longer exists. - `tasks/stream-processor-ergonomics.md` — targets the legacy hook-style processor API, replaced by the class-based StreamProcessor model. ### Deleted: historical logs - `apps/os/tasks/slack-google-auth-poc-implementation.md` — explicitly an "Implementation Log" (`state: done`), not actionable work; shipped in merged PR #1317. - `apps/os/tasks/stream-processor-class-design-notes.md` — design notes written alongside the class-based StreamProcessor migration, not a task. - `apps/os/tasks/workspace-codemode-implementation-log.md` — `state: done`, all 9 checkpoints ticked; the described work verifiably shipped on main. ### Kept but flagged for maintainer judgment - `tasks/cf-prd-orphaned-resources-cleanup.md`: Explicit not-in-scope follow-ups (preview account 376ef7ed cleanup, Doppler os-legacy-backup pruning) were never broken out into their own tasks; spin them out only if still wanted. - `tasks/codemode-capability-policy.md`: Still-unshipped, still-wanted design work, but duplicates `apps/os/tasks/codemode-capability-access-policy.md` and overlaps the active itx capability-system design notes — maintainer should consolidate into a single task. - `tasks/complete/2026-05-22-os-captun-worker-test-tunnel.md`: apps/os still depends on the unpublished pkg.pr.new/captun@14 build (the task's stated stopgap); a published captun/worker release would be a separate follow-up, not a reason to keep this file. - `tasks/dead-code-and-docs-cleanup-audit.md`: Residual from this audit: packages/iterate is still excluded from root build/typecheck/test (`--filter '!iterate'`); if that CI gap matters, open a fresh small task rather than keeping this stale inventory. - `tasks/doppler-shared-and-os-secrets-audit.md`: Audit still unrun and wanted, but needs a rewrite first: replace Clerk-key expectations with iterateAuth, point AppConfig refs at `apps/os/src/config.ts` (`app.ts` and `packages/shared/src/apps/config.ts` were deleted in PR #1411), and refresh the 2026-05-18 baseline. - `tasks/ignoreme-email-security.md`: If outbound email via Resend is ever reintroduced in the rebuilt apps/os, recipient allowlisting should be designed fresh against the itx/egress-secret-substitution layer, not this OS1-era plan. - `tasks/iterate-cli-distribution.md`: Live but ~90% of the file is OpenCode architecture research notes, not actionable steps; npm distribution already exists, so the remaining work (bun binary, brew, install script) should be restated as concrete tasks or the research trimmed. - `tasks/os-auth-spurious-logout-refresh.md`: PR #1410 left one open thread: a manual end-to-end "wait 5 minutes in prod" verification was never done, and the claims-staleness force-refresh was consciously skipped (≤30m propagation accepted) — file a new narrow task only if either still matters. - `tasks/os-deploy-time-jwks-fetch.md`: Code shipped in PR #1410; only remaining action is deleting `ITERATE_AUTH_JWKS` from Doppler os prd/preview (still present and shadowing the deploy-time fetch) — after that, delete this task. - `tasks/os-domain-capability-orpc-refactor-prd.md`: Sibling task `os-domain-capability-orpc-refactor-design.md` (its dependsOn target) is likely also completed and should be audited/deleted together. - `tasks/os-project-do-projection-reconciliation.md`: Scope item "rename IterateMcpServer to ProjectMcpServerConnection" is already done and could be ticked off; the rest is unshipped and still relevant. - `tasks/os-project-hostname-base-singular.md`: Scope file paths are stale post-PR #1411 (`app.ts`→`src/config.ts`, `sync-clerk-apps.ts`→`sync-auth-clients.ts`, `entry.workerd.ts` deleted, routing files moved to `src/ingress/`); task itself is still valid. - `tasks/os-project-route-authorization.md`: Still-wanted design work (referenced by live project-ingress-architecture task), but needs rewrite: Clerk OAuth and `ProjectMcpServerEntrypoint` references are dead — MCP moved off project ingress (410 stub) and auth is now apps/auth Principal-based. - `tasks/os-stream-runtime-big-refactors.md`: Only surviving idea: cosmetic no-compat rename of `events.iterate.com/...` event-type names (events app is deleted); re-file as a small standalone task if still wanted. - `apps/os/tasks/codemode-capability-access-policy.md`: Live work, but near-duplicates root-level `tasks/codemode-capability-policy.md` (same PR #1294); keep this copy and consolidate/delete the root one. - `apps/os/tasks/codemode-session-night-plan.md`: Open capability-scope questions from this plan live on in `codemode-capability-access-policy.md`; checkboxes are unticked but the work shipped via PRs #1294/#1305/#1402. - `apps/os/tasks/codemode-session-vertical-slice.md`: Last unchecked box (generalize self-callable bindings) shipped as the loopback-binding pattern used repo-wide; follow-on work lives in `codemode-session-night-plan.md`. - `apps/os/tasks/project-egress-and-secrets-architecture.md`: Design doc whose first vertical slice shipped (egress + secret substitution MVP); remaining secret-DO/policy/approval/OAuth design is still live but needs grooming: drop completed PoC sections, update Clerk-scope terminology, and reconcile with itx DECISIONS.md as the newer design-of-record for egress wiring. - `apps/os/tasks/project-egress-intercept-tunnel-latency.md`: Still-relevant latency work, but file refs are stale (`entry.workerd.ts` → `src/worker.ts`; vendored `apps/os/src/lib/captun` removed for the published captun package in #1361) and the benchmark numbers predate the #1411 worker rebuild — re-benchmark before picking an option. - `apps/os/tasks/project-ingress-architecture.md`: Live, actively-maintained ingress reference (edited today in #1416), but needs a refresh: Clerk auth sections, `Project.checkAccess`, and the streams-upstream proxy model are superseded (auth worker, principal claims, bundled project worker), and the 2026-05-05 status checklist is partly outdated. - `apps/os/tasks/stream-processor-class-migration-log.md`: Migration log (merged today via #1402, which links to it as the canonical rationale) — not an actionable task; contains unique I6-I8 forensics not in the PR body, consider moving to docs/ alongside `tasks/migration-notes/` rather than deleting. - `apps/os/tasks/stream-subscriber-delivery-refactor.md`: Core design shipped differently via the class-model cutover (#1401/#1402/#1394); only live remainder is migrating `codemode.streamEvents`, `StreamsCapability.stream()`, and project-mcp-server-connection off the OS-internal NDJSON shim in `new-stream-runtime.ts` — consider replacing this large draft with a small task for that. - `apps/os/tasks/workspace-codemode-implementation-log.md`: Done implementation log; only marginally unique note is the rationale that plain method objects (not class instances) cross DO RPC, which is now embodied in the shipped workspace DO code. - `apps/os/tasks/migration-notes/`: Historical migration logs (not tasks) committed with and cited by merged PR #1402 one day ago; contain unique per-domain decisions plus the legacy-subscriber gap behind the 2026-06-10 prd Slack outage — maintainer should relocate to docs/ or delete deliberately. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- > [!NOTE] > **Low Risk** > Documentation and task-file deletions only; no application runtime or API behavior changes in the diff. > > **Overview** > **Aligns OS documentation with the current worker, auth, routing, and itx reality**, and **removes a large set of completed or obsolete task files** from `apps/os/tasks/` and `tasks/`. > > The **README / AGENTS** and **`architecture-and-operations.md`** rewrites drop Clerk-era and deleted-entrypoint references (`src/app.ts`, `src/entry.workerd.ts`, `/org/:organizationSlug`) in favor of **`src/worker.ts`**, **Iterate Auth**, **project-scoped routes** (`/projects/...`, `/new-project`), **canonical MCP** (`APP_CONFIG_MCP__BASE_URL`, auth-worker OAuth), **itx** endpoints, and **`sync-auth-clients.ts`**. Deploy docs now distinguish ambient **`pnpm cf:deploy`** from production **`pnpm deploy`**. E2E docs point at **`pnpm e2e`** and **`pnpm e2e:itx`** instead of removed capnweb vitest configs. > > **Cap'n Web tombstones** in `iterate-context*.md` redirect readers to **itx** (`src/itx/`, `itx-spec.md`). Research notes get **historical** headers; **itx-spec** notes merged status on main and documents that **`connectItx` is one-shot** (no §6.3 reconnect loop). **itx README / `Stubify`** docs are corrected: typed caps use **`itx.cap("name") as Stubify<...>`**, not declaration merging. > > **CONTEXT.md** fixes the example that claimed org UI lived under `/org/...`. **headless-local-debugging** uses **`/new-project`**. > > **Task grooming** deletes many markdown tasks whose work is done, superseded (itx, auth worker), or OS1-dead — including codemode vertical-slice plans, domain oRPC refactor design, egress MVP, Slack processor unwind, and similar inventory items. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit a4f093f. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>   ## Environment Config Lease    Lease: `preview-5` Doppler config: `preview_5` Type: `environment-config-lease` Leased until: 2026-06-10T13:19:51.555Z ### OS Status: deployed Commit: `a4f093f` Preview: https://os.iterate-preview-5.com [Workflow run](https://github.com/iterate/iterate/actions/runs/27275677688) Updated: 2026-06-10T12:23:34.040Z  --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

prototype stream processor v2

dbdd2d6

jonastemplestein mentioned this pull request Jun 9, 2026

Class-model stream processors across apps/os: DO-hosted, callable subscriptions, legacy model deleted #1402

Merged

jonastemplestein added 2 commits June 9, 2026 13:32

add stream processor v2 migration log

3ca6dcf

migrate browser stream processors to v2

2116d85

jonastemplestein marked this pull request as ready for review June 9, 2026 13:38

cursor Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread packages/streams/src/browser/stream-browser-store.ts

clean up stream processor class migration

ab50389

jonastemplestein changed the title ~~[codex] prototype stream processor v2~~ [codex] migrate stream processors to class model Jun 9, 2026

simplify class-based stream processors

9b8460d

cursor Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread packages/streams/src/browser/stream-browser-store.ts

jonastemplestein and others added 6 commits June 9, 2026 17:02

enforce stream processor override typing

f0292dc

Merge remote-tracking branch 'origin/main' into suave-college

31b1cc5

# Conflicts: # packages/streams/example-app/src/routes/-event-feed-view.tsx # packages/streams/package.json

jonastemplestein mentioned this pull request Jun 9, 2026

[codex] stream processor: crisp batch model #1409

Merged

jonastemplestein and others added 4 commits June 9, 2026 22:14

jonastemplestein changed the title ~~[codex] migrate stream processors to class model~~ Class-based stream processors: crisp batch model, honest contracts, regression tests Jun 9, 2026

jonastemplestein and others added 2 commits June 9, 2026 22:29

Revert "stream processor: atomic browser checkpoints and leader shutd…

dc57d28

…own" This reverts commit 41762ad.

jonastemplestein merged commit 022ea6c into main Jun 9, 2026
7 of 8 checks passed

jonastemplestein deleted the suave-college branch June 9, 2026 21:55

jonastemplestein mentioned this pull request Jun 10, 2026

docs(os): fix AGENTS.md, rewrite architecture-and-operations, retire capnweb pointers, fix task states #1432

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Class-based stream processors: crisp batch model, honest contracts, regression tests#1401

Class-based stream processors: crisp batch model, honest contracts, regression tests#1401
jonastemplestein merged 17 commits into
mainfrom
suave-college

jonastemplestein commented Jun 9, 2026 •

edited by iterate-bot

Loading

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jonastemplestein commented Jun 9, 2026 • edited by iterate-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

The model

Contracts and type inference

Hosts

Tests

Validation

Environment Config Lease

OS

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jonastemplestein commented Jun 9, 2026 •

edited by iterate-bot

Loading