Local-First Automerge Migration Plan

Migrate the notebook from RPC-with-optimistic-UI to true local-first Automerge ownership.

Status

🎉 The migration is complete. The notebook is local-first. Cells mutate instantly in WASM, sync flows to the daemon via Automerge, execution works, widgets work. Everything below is post-migration cleanup and future improvements.

Problem

The current architecture maintains four copies of the notebook:

React useState ←→ invoke() ←→ NotebookState ←→ SyncClient AutoCommit ←→ Daemon AutoCommit
     (1)                          (2)                  (3)                     (4)

1. React cells state — useState<NotebookCell[]> in useNotebook.ts:286
2. NotebookState — nbformat::v4::Notebook struct in crates/notebook/src/notebook_state.rs:152
3. Sync client's AutoCommit — local Automerge replica in NotebookSyncClient (crates/runtimed/src/notebook_sync_client.rs:433)
4. Daemon's NotebookDoc — canonical Automerge doc in NotebookRoom (crates/runtimed/src/notebook_sync_server.rs:452)

Cell mutations like addCell and deleteCell use blocking RPC — the frontend awaits invoke(), which round-trips through copies (2), (3), and (4) before the frontend updates copy (1). The user pays full IPC + sync latency for operations that don't need backend involvement.

What's already right

• Outputs flow through Automerge, not RPC. Frontend onOutput is intentionally () => {} (App.tsx:271-276). Outputs arrive via notebook:updated Automerge sync.
• Daemon reads source from Automerge for execution. ExecuteCell reads from the doc (notebook_sync_server.rs:1850), not from whatever the frontend sent.
• Source uses Automerge Text CRDT. Character-level merging via update_text() (Myers diff).
• Python agents are full Automerge peers. The runtimed Python package (crates/runtimed-py) connects via the same Unix socket, holds its own AutoCommit doc, and syncs bidirectionally. Agents can create_cell, set_source, append_source, delete_cell — all as CRDT operations that merge with concurrent user edits.

What's wrong

• NotebookState is redundant. It shadows the sync client's Automerge doc. The receiver loop at lib.rs:420-424 overwrites it on every peer update.
• Frontend has no Automerge doc. It receives materialized CellSnapshot[] and can't do local CRDT mutations.
• addCell and deleteCell block on RPC for things the frontend can determine locally (UUID generation, "last cell" validation).
• Full-state replacement on sync. notebook:updated does setCells(newCells), clobbering any in-flight optimistic state, which forces the blocking RPC pattern.

Peers to account for

Python agents are first-class peers. An agent calling session.append_source(cell_id, text) while a user types in the same cell produces concurrent Text CRDT operations that Automerge merges automatically. The migration must preserve this — agents should never need to change their API.

Target architecture

┌──────────────────────────────┐
│  Frontend                    │
│  Local AutoCommit doc        │
│  React state derived from doc│
│  All cell CRUD is local      │
└──────────┬───────────────────┘
           │ Binary Automerge sync messages
           │ (Tauri events or custom protocol)
┌──────────▼───────────────────┐
│  Tauri Process (thin relay)  │
│  Forwards sync messages      │
│  Handles: save, format, OS   │
└──────────┬───────────────────┘
           │ Unix socket (v2 typed frames)
┌──────────▼───────────────────┐
│  Daemon                      │
│  Canonical AutoCommit doc    │
│  Writes: outputs, exec_count │
│  Kernel, env management      │
└──────────┬───────────────────┘
           │ Unix socket (v2 typed frames)
┌──────────▼───────────────────┐
│  Python Agent                │
│  Own AutoCommit doc          │
│  Writes: cells, source       │
│  Reads: outputs, status      │
└──────────────────────────────┘

All peers hold Automerge docs. Mutations are local-first. Sync is automatic and bidirectional.

const deleteCell = useCallback(async (cellId: string) => {
  try {
    await invoke("delete_cell", { cellId });
    setCells((prev) => prev.filter((c) => c.id !== cellId));
    setDirty(true);
  } catch (e) { ... }
}, []);

const deleteCell = useCallback((cellId: string) => {
  // Validate locally — don't delete the last cell
  setCells((prev) => {
    if (prev.length <= 1) return prev;
    return prev.filter((c) => c.id !== cellId);
  });
  setDirty(true);
  // Fire-and-forget sync to backend
  invoke("delete_cell", { cellId }).catch((e) =>
    logger.error("[notebook] delete_cell sync failed:", e)
  );
}, []);

const addCell = useCallback(async (cellType, afterCellId?) => {
  const newCell = await invoke<NotebookCell>("add_cell", { cellType, afterCellId });
  setCells((prev) => { /* insert newCell */ });
  ...
}, []);

const addCell = useCallback((cellType, afterCellId?) => {
  const cellId = crypto.randomUUID();
  const newCell: NotebookCell = {
    id: cellId,
    cell_type: cellType,
    source: "",
    outputs: [],
    execution_count: null,
  };
  setCells((prev) => { /* insert newCell at position */ });
  setFocusedCellId(cellId);
  setDirty(true);
  // Fire-and-forget — backend uses provided cellId
  invoke("add_cell", { cellId, cellType, afterCellId }).catch((e) =>
    logger.error("[notebook] add_cell sync failed:", e)
  );
  return newCell;
}, []);

Phase 2: Frontend Automerge doc — PR #553

Draft PR #547 was abandoned — it accumulated layers of fixes for the JS string→Text mismatch we didn't understand yet. PR #553 is a clean rewrite using runtimed-wasm.

Goal: The frontend owns a local Automerge document. All document mutations happen instantly on the local doc. React state is derived from it. The Tauri process becomes a sync relay.

Effort: Medium (1-2 weeks). Risk: Low — the WASM eliminates the type mismatch that blocked us. Widgets confirmed working.

Strategy: Feature flag toggle. Build useAutomergeNotebook alongside useNotebook, controlled by localStorage or ?automerge=true URL param. Fresh hook using NotebookHandle from runtimed-wasm — no @automerge/automerge JS dependency.

Status: ✅ Complete. useAutomergeNotebook is the only path. useNotebook.ts and useNotebookDispatch.ts removed. Widgets working.

End state vs pragmatic step: The WASM is the pragmatic unblock. The destination is the frontend owning a proper JS Automerge doc with presence, cursors, and ecosystem plugins (@automerge/codemirror). Once the WASM path is stable and the Tauri relay is simplified, switching back to @automerge/automerge JS with ImmutableString for non-text fields is a well-scoped change. We'll be off the broken path and into the paved one.

Hard-won lessons (from QA and debugging)

What to reuse from PR #547

The Rust-side relay infrastructure is sound and reusable. The JS hook needs a clean rewrite.

Keep (from PR #547's Rust changes):

• raw_sync_tx / connect_split_with_raw_sync() — daemon sync message forwarding
• get_automerge_doc_bytes command — exports Tauri-side doc as bytes
• send_automerge_sync command — receives frontend sync messages
• frontend_peer_state — separate sync state for the frontend peer (deferred init on GetDocBytes)
• automerge:from-daemon Tauri event — relay to frontend

Rewrite (fresh hook using runtimed-wasm) ✅:

• useAutomergeNotebook — loads NotebookHandle from WASM, not @automerge/automerge
• Cell mutations: handle.add_cell(), handle.delete_cell(), handle.update_source() — no Automerge.change(), no proxy methods, no RawString
• Sync: handle.generate_sync_message() / handle.receive_sync_message() — same relay, different WASM
• Materialization: handle.get_cells_json() → parse → cellSnapshotsToNotebookCells() → React state
• Shared CellSnapshot type extracted to materialize-cells.ts (used by both old and new hooks during migration)
• Widgets working — ipywidgets comm messages flow correctly through the new path

Dropped (bandaid ripped):

• useNotebook.ts — deleted, useAutomergeNotebook is the only hook
• useNotebookDispatch.ts — deleted, no more feature flag toggle
• @automerge/automerge npm dependency — no longer needed
• All RawString / ImmutableString / proxy method workarounds

Sub-PR 2D — Post-migration cleanup — PR #554

Done in PR #554:

• Remove useNotebook.ts — deleted (PR feat(notebook): add useAutomergeNotebook hook via runtimed-wasm #553)
• Remove useNotebookDispatch.ts — deleted (PR feat(notebook): add useAutomergeNotebook hook via runtimed-wasm #553)
• Remove feature flag infrastructure (PR feat(notebook): add useAutomergeNotebook hook via runtimed-wasm #553)
• Remove dead Tauri commands: load_notebook, update_cell_source, add_cell, delete_cell, refresh_from_automerge (−217 lines Rust)
• Remove cell_snapshot_to_frontend helper
• Remove notebook:updated cell emissions from init and receiver loop (metadata emissions kept — 3 active listeners)

Still to do (next PR):

Output streaming: onOutput stays no-opped

onOutput: () => {} — wiring appendOutput causes duplicate outputs. Broadcast appends an output, then Automerge sync replaces all cells with doc state that already has the output. Without a stable output ID to deduplicate, both appear.

Future fix: Add a sequence number or content hash to broadcast output messages so the frontend can skip outputs already present in Automerge state. This is a new feature, not a migration task.

Build pipeline note

crates/runtimed-wasm/src/notebook_doc.rs is currently a copy of crates/runtimed/src/notebook_doc.rs with daemon-only methods removed. When notebook_doc.rs changes (rare — usually daemon-side output operations), the WASM copy must be updated and wasm-pack build re-run. Future cleanup: extract a shared notebook-doc crate that both depend on.

Future improvements (post-migration)

Phase 3: Authority boundary hardening

Goal: Formalize which fields each peer can write, so CRDT convergence matches user expectations.

Effort: Small (days). Risk: Low.

Writer roles

Conflict scenarios

clear_outputs ownership

Today the frontend clears outputs locally (clearCellOutputs in useNotebook.ts:441) AND the daemon clears via Automerge. In the new model:

• Daemon owns output clearing — it clears when execution starts
• Frontend stops rendering stale outputs when it sees execution_started broadcast
• Remove frontend-side clearCellOutputs mutation on the Automerge doc
• Frontend can show a visual indicator (dimming) while waiting for new outputs

Phase 4: Optimize Tauri sync relay (optional, future)

Goal: Make the Tauri relay as thin and fast as possible. Tauri stays in the path — the frontend should never speak directly to the Unix socket for security reasons.

Effort: Small. Risk: Low.

Why Tauri stays in the loop

The daemon Unix socket is unauthenticated — any process that can reach it can read/write any notebook. In production this is fine because only local processes connect. But letting the webview open a raw socket (even localhost WebSocket) would let a compromised renderer talk to the daemon without Tauri's mediation. Tauri should own the connection lifecycle: auth, handshake, connection teardown on window close.

Binary IPC precedent

We already pass binary data through Tauri events for widget buffers (useDaemonKernel.ts:316-321) — number[][] round-tripped via JSON. This works but is not ideal for Automerge sync messages which are opaque binary blobs that don't benefit from JSON encoding.

Optimization options

Recommendation: start with base64-in-events in Phase 2 (simplest, unblocks everything), then migrate to Channel<Vec<u8>> if profiling shows encoding overhead matters.

The blob store already has its own HTTP server (http://127.0.0.1:{blobPort}/blob/{hash}) for large output payloads, so sync messages only carry manifest hashes, not raw output data. This keeps sync message sizes small.

• Benchmark sync message sizes in practice (expect sub-KB for source edits, larger for cell add with initial content)
• If base64 overhead is measurable: migrate to Tauri Channel<Vec<u8>> for binary streaming
• Profile end-to-end latency: frontend Automerge change → Tauri relay → daemon apply → broadcast → other peer receives

Key files reference

Frontend

Tauri process (crates/notebook)

Daemon (crates/runtimed)

Python agent

Non-goals

• Frontend direct access to the daemon socket. The daemon's Unix socket is unauthenticated — any process that connects can read/write any notebook room. Tauri must mediate all daemon communication so the webview renderer never holds a raw socket. This is a security boundary, not a performance optimization to remove later.
• Replacing the daemon's Automerge with automerge-repo. The daemon's custom sync protocol (v2 typed frames multiplexing sync + requests + broadcasts) is well-suited to the single-doc-per-room model. automerge-repo is designed for multi-doc repos with discovery — unnecessary complexity here.
• Moving Automerge to the Python agent. The Python agent already has a full Automerge peer via the Rust NotebookSyncClient. No JS Automerge needed on the Python side.
• Real-time collaborative cursors. Desirable but separate concern. Can be implemented later via Automerge ephemeral messages or the existing broadcast channel.
• Operational transform for CodeMirror. The @automerge/codemirror plugin exists but is a Phase 2+ consideration. Initial implementation can use updateText on every change event.