Skip to content

Commit cc5c691

Browse files
TakhoffmanTakhoffman
authored
feat(ui): render assistant directives and add embed tag (#64104)
* Add embed rendering for Control UI assistant output * Add changelog entry for embed rendering * Harden canvas path resolution and stage isolation * Secure assistant media route and preserve UI avatar override * Fix chat media and history regressions * Harden embed iframe URL handling * Fix embed follow-up review regressions * Restore offloaded chat attachment persistence * Harden hook and media routing * Fix embed review follow-ups * feat(ui): add configurable embed sandbox mode * fix(gateway): harden assistant media and auth rotation * fix(gateway): restore websocket pairing handshake flows * fix(gateway): restore ws hello policy details * Restore dropped control UI shell wiring * Fix control UI reconnect cleanup regressions * fix(gateway): restore media root and auth getter compatibility * feat(ui): rename public canvas tag to embed * fix(ui): address remaining media and gateway review issues * fix(ui): address remaining embed and attachment review findings * fix(ui): restore stop control and tool card inputs * fix(ui): address history and attachment review findings * fix(ui): restore prompt contribution wiring * fix(ui): address latest history and directive reviews * fix(ui): forward password auth for assistant media * fix(ui): suppress silent transcript tokens with media * feat(ui): add granular embed sandbox modes * fix(ui): preserve relative media directives in history * docs(ui): document embed sandbox modes * fix(gateway): restrict canvas history hoisting to tool entries * fix(gateway): tighten embed follow-up review fixes * fix(ci): repair merged branch type drift * fix(prompt): restore stable runtime prompt rendering * fix(ui): harden local attachment preview checks * fix(prompt): restore channel-aware approval guidance * fix(gateway): enforce auth rotation and media cleanup * feat(ui): gate external embed urls behind config * fix(ci): repair rebased branch drift * fix(ci): resolve remaining branch check failures
1 parent d83a85c commit cc5c691

75 files changed

Lines changed: 8114 additions & 1783 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,13 @@ Docs: https://docs.openclaw.ai
3737
- Gateway: add a `commands.list` RPC so remote gateway clients can discover runtime-native, text, skill, and plugin commands with surface-aware naming and serialized argument metadata. (#62656) Thanks @samzong.
3838
- Models/providers: add per-provider `models.providers.*.request.allowPrivateNetwork` for trusted self-hosted OpenAI-compatible endpoints, keep the opt-in scoped to model request surfaces, and refresh cached WebSocket managers when request transport overrides change. (#63671) Thanks @qas.
3939
- Feishu: standardize request user agents and register the bot as an AI agent so Feishu deployments identify OpenClaw consistently. (#63835) Thanks @evandance.
40+
- Docs i18n: chunk raw doc translation, reject truncated tagged outputs, avoid ambiguous body-only wrapper unwrapping, and recover from terminated Pi translation sessions without changing the default `openai/gpt-5.4` path. (#62969, #63808) Thanks @hxy91819.
41+
- Gateway: split startup and runtime seams so gateway lifecycle sequencing, reload state, and shutdown behavior stay easier to maintain without changing observed behavior. (#63975) Thanks @gumadeiras.
42+
- Control UI/webchat: normalize assistant `MEDIA:`/reply/voice directives into structured bubble rendering, rename the unreleased rich web shortcode to `[embed ...]`, and surface session runtime roots so hosted web content is written to the correct document path instead of guessed local files.
4043
- Matrix/partial streaming: add MSC4357 live markers to draft preview sends and edits so supporting Matrix clients can render a live/typewriter animation and stop it when the final edit lands. (#63513) Thanks @TigerInYourDream.
4144
- Control UI/dreaming: simplify the Scene and Diary surfaces, preserve unknown phase state for partial status payloads, and stabilize waiting-entry recency ordering so Dreaming status and review lists stay clear and deterministic. (#64035) Thanks @davemorin.
4245
- Agents: add an opt-in strict-agentic embedded Pi execution contract for GPT-5-family runs so plan-only or filler turns keep acting until they hit a real blocker. (#64241) Thanks @100yenadmin.
4346
- Agents/OpenAI: add provider-owned OpenAI/Codex tool schema compatibility and surface embedded-run replay/liveness state for long-running runs. (#64300) Thanks @100yenadmin.
44-
- Docs i18n: chunk raw doc translation, reject truncated tagged outputs, avoid ambiguous body-only wrapper unwrapping, and recover from terminated Pi translation sessions without changing the default `openai/gpt-5.4` path. (#62969, #63808) Thanks @hxy91819.
4547
- Dreaming/memory-wiki: add ChatGPT import ingestion plus new `Imported Insights` and `Memory Palace` diary subtabs so Dreaming can inspect imported source chats, compiled wiki pages, and full source pages directly from the UI. (#64505)
4648

4749
### Fixes

‎docs/gateway/configuration-reference.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2895,6 +2895,8 @@ See [Plugins](/tools/plugin).
28952895
enabled: true,
28962896
basePath: "/openclaw",
28972897
// root: "dist/control-ui",
2898+
// embedSandbox: "scripts", // strict | scripts | trusted
2899+
// allowExternalEmbedUrls: false, // dangerous: allow absolute external http(s) embed URLs
28982900
// allowedOrigins: ["https://control.example.com"], // required for non-loopback Control UI
28992901
// dangerouslyAllowHostHeaderOriginFallback: false, // dangerous Host-header origin fallback mode
29002902
// allowInsecureAuth: false,

‎docs/reference/rich-output-protocol.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Rich Output Protocol
2+
3+
Assistant output can carry a small set of delivery/render directives:
4+
5+
- `MEDIA:` for attachment delivery
6+
- `[[audio_as_voice]]` for audio presentation hints
7+
- `[[reply_to_current]]` / `[[reply_to:<id>]]` for reply metadata
8+
- `[embed ...]` for Control UI rich rendering
9+
10+
These directives are separate. `MEDIA:` and reply/voice tags remain delivery metadata; `[embed ...]` is the web-only rich render path.
11+
12+
## `[embed ...]`
13+
14+
`[embed ...]` is the only agent-facing rich render syntax for the Control UI.
15+
16+
Self-closing example:
17+
18+
```text
19+
[embed ref="cv_123" title="Status" /]
20+
```
21+
22+
Rules:
23+
24+
- `[view ...]` is no longer valid for new output.
25+
- Embed shortcodes render in the assistant message surface only.
26+
- Only URL-backed embeds are rendered. Use `ref="..."` or `url="..."`.
27+
- Block-form inline HTML embed shortcodes are not rendered.
28+
- The web UI strips the shortcode from visible text and renders the embed inline.
29+
- `MEDIA:` is not an embed alias and should not be used for rich embed rendering.
30+
31+
## Stored Rendering Shape
32+
33+
The normalized/stored assistant content block is a structured `canvas` item:
34+
35+
```json
36+
{
37+
"type": "canvas",
38+
"preview": {
39+
"kind": "canvas",
40+
"surface": "assistant_message",
41+
"render": "url",
42+
"viewId": "cv_123",
43+
"url": "/__openclaw__/canvas/documents/cv_123/index.html",
44+
"title": "Status",
45+
"preferredHeight": 320
46+
}
47+
}
48+
```
49+
50+
Stored/rendered rich blocks use this `canvas` shape directly. `present_view` is not recognized.

‎docs/web/control-ui.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,38 @@ Cron jobs panel notes:
138138
- Gateway persists aborted partial assistant text into transcript history when buffered output exists
139139
- Persisted entries include abort metadata so transcript consumers can tell abort partials from normal completion output
140140

141+
## Hosted embeds
142+
143+
Assistant messages can render hosted web content inline with the `[embed ...]`
144+
shortcode. The iframe sandbox policy is controlled by
145+
`gateway.controlUi.embedSandbox`:
146+
147+
- `strict`: disables script execution inside hosted embeds
148+
- `scripts`: allows interactive embeds while keeping origin isolation; this is
149+
the default and is usually enough for self-contained browser games/widgets
150+
- `trusted`: adds `allow-same-origin` on top of `allow-scripts` for same-site
151+
documents that intentionally need stronger privileges
152+
153+
Example:
154+
155+
```json5
156+
{
157+
gateway: {
158+
controlUi: {
159+
embedSandbox: "scripts",
160+
},
161+
},
162+
}
163+
```
164+
165+
Use `trusted` only when the embedded document genuinely needs same-origin
166+
behavior. For most agent-generated games and interactive canvases, `scripts` is
167+
the safer choice.
168+
169+
Absolute external `http(s)` embed URLs stay blocked by default. If you
170+
intentionally want `[embed url="https://..."]` to load third-party pages, set
171+
`gateway.controlUi.allowExternalEmbedUrls: true`.
172+
141173
## Tailnet access (recommended)
142174

143175
### Integrated Tailscale Serve (preferred)

‎src/agents/pi-embedded-runner/system-prompt.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ export function buildEmbeddedSystemPrompt(params: {
4343
channel?: string;
4444
/** Supported message actions for the current channel (e.g., react, edit, unsend) */
4545
channelActions?: string[];
46+
canvasRootDir?: string;
4647
};
4748
messageToolHints?: string[];
4849
sandboxInfo?: EmbeddedSandboxInfo;

‎src/agents/system-prompt-params.test.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import os from "node:os";
33
import path from "node:path";
44
import { describe, expect, it } from "vitest";
55
import type { OpenClawConfig } from "../config/config.js";
6+
import { resolveStateDir } from "../config/paths.js";
67
import { buildSystemPromptParams } from "./system-prompt-params.js";
78

89
async function makeTempDir(label: string): Promise<string> {
@@ -101,4 +102,12 @@ describe("buildSystemPromptParams repo root", () => {
101102

102103
expect(runtimeInfo.repoRoot).toBeUndefined();
103104
});
105+
106+
it("includes the default profile canvas root in runtimeInfo", async () => {
107+
const workspaceDir = await makeTempDir("canvas-root");
108+
109+
const { runtimeInfo } = buildParams({ workspaceDir });
110+
111+
expect(runtimeInfo.canvasRootDir).toBe(path.resolve(path.join(resolveStateDir(), "canvas")));
112+
});
104113
});

‎src/agents/system-prompt-params.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
import fs from "node:fs";
22
import path from "node:path";
3+
import { resolveStateDir } from "../config/paths.js";
34
import type { OpenClawConfig } from "../config/types.openclaw.js";
45
import { findGitRoot } from "../infra/git-root.js";
6+
import { resolveHomeRelativePath } from "../infra/home-dir.js";
57
import {
68
formatUserTime,
79
resolveUserTimeFormat,
@@ -23,6 +25,7 @@ export type RuntimeInfoInput = {
2325
/** Supported message actions for the current channel (e.g., react, edit, unsend) */
2426
channelActions?: string[];
2527
repoRoot?: string;
28+
canvasRootDir?: string;
2629
};
2730

2831
export type SystemPromptRuntimeParams = {
@@ -47,18 +50,36 @@ export function buildSystemPromptParams(params: {
4750
const userTimezone = resolveUserTimezone(params.config?.agents?.defaults?.userTimezone);
4851
const userTimeFormat = resolveUserTimeFormat(params.config?.agents?.defaults?.timeFormat);
4952
const userTime = formatUserTime(new Date(), userTimezone, userTimeFormat);
53+
const stateDir = resolveStateDir(process.env);
54+
const canvasRootDir = resolveCanvasRootDir({
55+
config: params.config,
56+
stateDir,
57+
});
5058
return {
5159
runtimeInfo: {
5260
agentId: params.agentId,
5361
...params.runtime,
5462
repoRoot,
63+
canvasRootDir,
5564
},
5665
userTimezone,
5766
userTime,
5867
userTimeFormat,
5968
};
6069
}
6170

71+
function resolveCanvasRootDir(params: { config?: OpenClawConfig; stateDir: string }): string {
72+
const configured = params.config?.canvasHost?.root?.trim();
73+
if (configured) {
74+
return path.resolve(
75+
resolveHomeRelativePath(configured, {
76+
env: process.env,
77+
}),
78+
);
79+
}
80+
return path.resolve(path.join(params.stateDir, "canvas"));
81+
}
82+
6283
function resolveRepoRoot(params: {
6384
config?: OpenClawConfig;
6485
workspaceDir?: string;

0 commit comments

Comments
 (0)