Skip to content

Commit 78eb92e

Browse files
pashpashpashvincentkoc
pashpashpash
and
vincentkoc
authored
Route Codex message tool replies back to WebChat and TUI (#81586)
* fix: route internal ui message tool replies * docs: document reserved codex sdk helpers * test(gateway): stabilize sessions send agent assertion * fix(agents): preserve rich internal source replies --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
1 parent faa443a commit 78eb92e

23 files changed

Lines changed: 765 additions & 140 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Docs: https://docs.openclaw.ai
1515
- Agents: preserve source-reply delivery metadata when merging tool-returned media into the final reply, keeping message-tool-only replies deliverable and mirrored. Thanks @pashpashpash and @vincentkoc.
1616
- macOS/companion: require system TLS trust before pinning a first-use direct `wss://` gateway certificate and honor `gateway.remote.tlsFingerprint` as the explicit pin for remote node-mode sessions, so fresh endpoints fail closed when macOS cannot trust the certificate unless configured out of band. Fixes #50642. Thanks @BunsDev.
1717
- Update: snapshot config before update-time repair and restart writes, preserve plugin install records through doctor cleanup, and block unsafe config size drops while an update is in progress. Fixes #80077. (#80257) Thanks @Jerry-Xin and @vincentkoc.
18+
- WebChat/TUI: route Codex `tools.message` source replies to the active internal UI turn and mirror them to session history, so message-tool-only harness replies, including rich presentation and button-only replies, no longer disappear while WebChat and TUI remain non-targetable outbound channels. (#81586) Thanks @pashpashpash.
1819
- Sessions/status: classify ACP spawn-child sessions as `kind: "spawn-child"` instead of `"direct"` in `openclaw sessions` and status output; extract the duplicated session-kind classifier into a shared helper (`src/sessions/classify-session-kind.ts`) so both surfaces stay in sync. Fixes catalog #19. (#79544)
1920
- Sessions/Gateway: report `agentRuntime.id: "acpx"` (or stored backend id) with `source: "session-key"` for ACP control-plane session rows in `openclaw sessions --json`, `openclaw status`, and Gateway session RPC responses instead of the incorrect `"auto"` / `"pi"` implicit fallback. Fixes catalog #18. (#79550)
2021
- Telegram: delete tool-progress-only draft bubbles before rotating to the real answer, preventing orphaned progress messages in streamed replies.

‎docs/plugins/sdk-subpaths.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -363,15 +363,17 @@ focused channel/runtime subpaths, `config-contracts`, `string-coerce-runtime`,
363363
</Accordion>
364364

365365
<Accordion title="Reserved bundled-helper subpaths">
366-
These private compatibility surfaces are reserved for their owning bundled
367-
plugin. New reusable host contracts should use generic SDK subpaths such as
368-
`plugin-sdk/gateway-runtime`, `plugin-sdk/security-runtime`, and
366+
Reserved bundled-helper SDK subpaths are narrow owner-specific surfaces for
367+
bundled plugin code. They are tracked in the SDK inventory so package
368+
builds and aliasing stay deterministic, but they are not general plugin
369+
authoring APIs. New reusable host contracts should use generic SDK subpaths
370+
such as `plugin-sdk/gateway-runtime`, `plugin-sdk/security-runtime`, and
369371
`plugin-sdk/plugin-config-runtime`.
370372

371-
| Subpath | Key exports |
373+
| Subpath | Owner and purpose |
372374
| --- | --- |
373-
| `plugin-sdk/codex-mcp-projection` | Codex-owned user MCP server projection helper for the bundled Codex app-server harness |
374-
| `plugin-sdk/codex-native-task-runtime` | Codex-owned detached task runtime helpers for native subagent mirroring |
375+
| `plugin-sdk/codex-mcp-projection` | Bundled Codex plugin helper for projecting user MCP server config into Codex app-server thread config |
376+
| `plugin-sdk/codex-native-task-runtime` | Bundled Codex plugin helper for mirroring Codex app-server native subagents into OpenClaw task state |
375377

376378
</Accordion>
377379
</AccordionGroup>

‎docs/web/tui.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ Notes:
7373
## Sending + delivery
7474

7575
- Messages are sent to the Gateway; delivery to providers is off by default.
76+
- The TUI is an internal source surface like WebChat, not a generic outbound channel. Harnesses that require `tools.message` for visible replies can satisfy the active TUI turn with a targetless `message.send`; explicit provider delivery still uses normal configured channels and never falls back to `lastChannel`.
7677
- Turn delivery on:
7778
- `/deliver on`
7879
- or the Settings panel

‎docs/web/webchat.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ WebChat has two separate data paths:
5151

5252
- The session JSONL file is the durable model/runtime transcript. For normal agent runs, Pi persists model-visible `user`, `assistant`, and `toolResult` messages through its session manager. WebChat does not write arbitrary delivery, status, or helper text into that transcript.
5353
- Gateway `ReplyPayload` events are the live delivery projection. They can be normalized for WebChat/channel display, block streaming, directive tags, media embedding, TTS/audio flags, and UI fallback behavior. They are not themselves the canonical session log.
54+
- Harnesses that require visible replies through `tools.message` still use WebChat as a current-run internal source reply sink. A targetless `message.send` from that active WebChat run is projected into the same chat and mirrored to the session transcript; WebChat does not become a reusable outbound channel and never inherits `lastChannel`.
5455
- WebChat injects assistant transcript entries only when the Gateway owns a displayed message outside a normal Pi assistant turn: `chat.inject`, non-agent command replies, aborted partial output, and WebChat-managed media transcript supplements.
5556
- `chat.history` reads the stored session transcript and applies WebChat display projection. If live assistant text appears during a run but disappears after history reload, first check whether the raw JSONL contains the assistant text, then whether `chat.history` projection stripped it, then whether the Control UI optimistic-tail merge replaced local delivery state with the persisted snapshot.
5657

‎extensions/codex/src/app-server/dynamic-tools.test.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,37 @@ describe("createCodexDynamicToolBridge", () => {
320320
]);
321321
});
322322

323+
it("records internal UI source replies separately from outbound messaging evidence", async () => {
324+
const toolResult = textToolResult("Sent to current chat.", {
325+
status: "ok",
326+
deliveryStatus: "sent",
327+
sourceReplySink: "internal-ui",
328+
sourceReply: {
329+
text: "visible reply",
330+
mediaUrls: ["/tmp/reply.png"],
331+
},
332+
});
333+
const bridge = createBridgeWithToolResult("message", toolResult);
334+
335+
const result = await handleMessageToolCall(bridge, {
336+
action: "send",
337+
message: "<think>private</think>visible reply",
338+
});
339+
340+
expect(result).toEqual(expectInputText("Sent to current chat."));
341+
expect(bridge.telemetry.didSendViaMessagingTool).toBe(true);
342+
expect(bridge.telemetry.messagingToolSentTexts).toEqual([]);
343+
expect(bridge.telemetry.messagingToolSentMediaUrls).toEqual([]);
344+
expect(bridge.telemetry.messagingToolSentTargets).toEqual([]);
345+
expect(bridge.telemetry.messagingToolSourceReplyPayloads).toEqual([
346+
{
347+
text: "visible reply",
348+
mediaUrl: "/tmp/reply.png",
349+
mediaUrls: ["/tmp/reply.png"],
350+
},
351+
]);
352+
});
353+
323354
it("does not record messaging side effects when the send fails", async () => {
324355
const tool = createTool({
325356
name: "message",

‎extensions/codex/src/app-server/dynamic-tools.ts‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import {
1515
type AnyAgentTool,
1616
type HeartbeatToolResponse,
1717
type MessagingToolSend,
18+
type MessagingToolSourceReplyPayload,
1819
wrapToolWithBeforeToolCallHook,
1920
} from "openclaw/plugin-sdk/agent-harness-runtime";
2021
import type { CodexDynamicToolsLoading } from "./config.js";
@@ -47,6 +48,7 @@ export type CodexDynamicToolBridge = {
4748
messagingToolSentTexts: string[];
4849
messagingToolSentMediaUrls: string[];
4950
messagingToolSentTargets: MessagingToolSend[];
51+
messagingToolSourceReplyPayloads: MessagingToolSourceReplyPayload[];
5052
heartbeatToolResponse?: HeartbeatToolResponse;
5153
toolMediaUrls: string[];
5254
toolAudioAsVoice: boolean;
@@ -77,6 +79,7 @@ export function createCodexDynamicToolBridge(params: {
7779
messagingToolSentTexts: [],
7880
messagingToolSentMediaUrls: [],
7981
messagingToolSentTargets: [],
82+
messagingToolSourceReplyPayloads: [],
8083
toolMediaUrls: [],
8184
toolAudioAsVoice: false,
8285
};
@@ -279,6 +282,11 @@ function collectToolTelemetry(params: {
279282
return;
280283
}
281284
params.telemetry.didSendViaMessagingTool = true;
285+
const sourceReplyPayload = extractInternalSourceReplyPayload(params.result?.details);
286+
if (sourceReplyPayload) {
287+
params.telemetry.messagingToolSourceReplyPayloads.push(sourceReplyPayload);
288+
return;
289+
}
282290
const text = readFirstString(params.args, ["text", "message", "body", "content"]);
283291
if (text) {
284292
params.telemetry.messagingToolSentTexts.push(text);
@@ -296,6 +304,41 @@ function collectToolTelemetry(params: {
296304
});
297305
}
298306

307+
function extractInternalSourceReplyPayload(
308+
details: unknown,
309+
): MessagingToolSourceReplyPayload | undefined {
310+
if (!isRecord(details) || details.sourceReplySink !== "internal-ui") {
311+
return undefined;
312+
}
313+
const rawPayload = details.sourceReply;
314+
if (!isRecord(rawPayload)) {
315+
return undefined;
316+
}
317+
const text = readFirstString(rawPayload, ["text", "message"]);
318+
const mediaUrls = collectMediaUrls(rawPayload);
319+
const mediaUrl =
320+
typeof rawPayload.mediaUrl === "string" && rawPayload.mediaUrl.trim()
321+
? rawPayload.mediaUrl.trim()
322+
: mediaUrls[0];
323+
const payload: MessagingToolSourceReplyPayload = {
324+
...(text ? { text } : {}),
325+
...(mediaUrl ? { mediaUrl } : {}),
326+
...(mediaUrls.length > 0 ? { mediaUrls } : {}),
327+
...(rawPayload.audioAsVoice === true ? { audioAsVoice: true } : {}),
328+
...(isRecord(rawPayload.presentation)
329+
? { presentation: rawPayload.presentation as never }
330+
: {}),
331+
...(isRecord(rawPayload.interactive) ? { interactive: rawPayload.interactive as never } : {}),
332+
...(isRecord(rawPayload.channelData) ? { channelData: rawPayload.channelData } : {}),
333+
...(typeof details.idempotencyKey === "string" && details.idempotencyKey.trim()
334+
? { idempotencyKey: details.idempotencyKey.trim() }
335+
: {}),
336+
};
337+
return text || mediaUrls.length > 0 || payload.presentation || payload.interactive
338+
? payload
339+
: undefined;
340+
}
341+
299342
function isRecord(value: unknown): value is Record<string, unknown> {
300343
return value !== null && typeof value === "object" && !Array.isArray(value);
301344
}

‎extensions/codex/src/app-server/event-projector.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import {
1717
type EmbeddedRunAttemptResult,
1818
type HeartbeatToolResponse,
1919
type MessagingToolSend,
20+
type MessagingToolSourceReplyPayload,
2021
type ToolProgressDetailMode,
2122
} from "openclaw/plugin-sdk/agent-harness-runtime";
2223
import { emitTrustedDiagnosticEvent } from "openclaw/plugin-sdk/diagnostic-runtime";
@@ -46,6 +47,7 @@ export type CodexAppServerToolTelemetry = {
4647
messagingToolSentTexts: string[];
4748
messagingToolSentMediaUrls: string[];
4849
messagingToolSentTargets: MessagingToolSend[];
50+
messagingToolSourceReplyPayloads?: MessagingToolSourceReplyPayload[];
4951
heartbeatToolResponse?: HeartbeatToolResponse;
5052
toolMediaUrls?: string[];
5153
toolAudioAsVoice?: boolean;
@@ -320,6 +322,7 @@ export class CodexAppServerEventProjector {
320322
messagingToolSentTexts: toolTelemetry.messagingToolSentTexts,
321323
messagingToolSentMediaUrls: toolTelemetry.messagingToolSentMediaUrls,
322324
messagingToolSentTargets: toolTelemetry.messagingToolSentTargets,
325+
messagingToolSourceReplyPayloads: toolTelemetry.messagingToolSourceReplyPayloads ?? [],
323326
heartbeatToolResponse: toolTelemetry.heartbeatToolResponse,
324327
toolMediaUrls: this.buildToolMediaUrls(toolTelemetry),
325328
toolAudioAsVoice: toolTelemetry.toolAudioAsVoice,

‎extensions/codex/src/app-server/run-attempt.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1899,6 +1899,7 @@ function buildCodexTurnStartFailureResult(params: {
18991899
messagingToolSentTexts: [],
19001900
messagingToolSentMediaUrls: [],
19011901
messagingToolSentTargets: [],
1902+
messagingToolSourceReplyPayloads: [],
19021903
cloudCodeAssistFormatError: false,
19031904
replayMetadata: {
19041905
hadPotentialSideEffects: false,

‎src/agents/pi-embedded-messaging.types.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import type { ReplyPayload } from "../auto-reply/reply-payload.js";
2+
13
export type MessagingToolSend = {
24
tool: string;
35
provider: string;
@@ -7,3 +9,16 @@ export type MessagingToolSend = {
79
text?: string;
810
mediaUrls?: string[];
911
};
12+
13+
export type MessagingToolSourceReplyPayload = Pick<
14+
ReplyPayload,
15+
| "audioAsVoice"
16+
| "channelData"
17+
| "interactive"
18+
| "mediaUrl"
19+
| "mediaUrls"
20+
| "presentation"
21+
| "text"
22+
> & {
23+
idempotencyKey?: string;
24+
};

‎src/agents/pi-embedded-runner/run.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,9 @@ function normalizeEmbeddedRunAttemptResult(
226226
messagingToolSentTexts?: EmbeddedRunAttemptForRunner["messagingToolSentTexts"] | null;
227227
messagingToolSentMediaUrls?: EmbeddedRunAttemptForRunner["messagingToolSentMediaUrls"] | null;
228228
messagingToolSentTargets?: EmbeddedRunAttemptForRunner["messagingToolSentTargets"] | null;
229+
messagingToolSourceReplyPayloads?:
230+
| EmbeddedRunAttemptForRunner["messagingToolSourceReplyPayloads"]
231+
| null;
229232
itemLifecycle?: EmbeddedRunAttemptForRunner["itemLifecycle"] | null;
230233
};
231234
return {
@@ -236,6 +239,7 @@ function normalizeEmbeddedRunAttemptResult(
236239
messagingToolSentTexts: raw.messagingToolSentTexts ?? [],
237240
messagingToolSentMediaUrls: raw.messagingToolSentMediaUrls ?? [],
238241
messagingToolSentTargets: raw.messagingToolSentTargets ?? [],
242+
messagingToolSourceReplyPayloads: raw.messagingToolSourceReplyPayloads ?? [],
239243
itemLifecycle: raw.itemLifecycle ?? {
240244
startedCount: 0,
241245
completedCount: 0,
@@ -2495,6 +2499,10 @@ export async function runEmbeddedPiAgent(
24952499
suppressToolErrorWarnings: params.suppressToolErrorWarnings,
24962500
inlineToolResultsAllowed: false,
24972501
didSendViaMessagingTool: attempt.didSendViaMessagingTool,
2502+
messagingToolSourceReplyPayloads: attempt.messagingToolSourceReplyPayloads,
2503+
sourceReplyDeliveryMode: params.sourceReplyDeliveryMode,
2504+
agentId: params.agentId,
2505+
runId: params.runId,
24982506
didSendDeterministicApprovalPrompt: attempt.didSendDeterministicApprovalPrompt,
24992507
heartbeatToolResponse: attempt.heartbeatToolResponse,
25002508
});
@@ -2571,6 +2579,7 @@ export async function runEmbeddedPiAgent(
25712579
messagingToolSentTexts: attempt.messagingToolSentTexts,
25722580
messagingToolSentMediaUrls: attempt.messagingToolSentMediaUrls,
25732581
messagingToolSentTargets: attempt.messagingToolSentTargets,
2582+
messagingToolSourceReplyPayloads: attempt.messagingToolSourceReplyPayloads,
25742583
heartbeatToolResponse: attempt.heartbeatToolResponse,
25752584
successfulCronAdds: attempt.successfulCronAdds,
25762585
};
@@ -2788,6 +2797,7 @@ export async function runEmbeddedPiAgent(
27882797
messagingToolSentTexts: attempt.messagingToolSentTexts,
27892798
messagingToolSentMediaUrls: attempt.messagingToolSentMediaUrls,
27902799
messagingToolSentTargets: attempt.messagingToolSentTargets,
2800+
messagingToolSourceReplyPayloads: attempt.messagingToolSourceReplyPayloads,
27912801
heartbeatToolResponse: attempt.heartbeatToolResponse,
27922802
successfulCronAdds: attempt.successfulCronAdds,
27932803
};
@@ -2839,6 +2849,7 @@ export async function runEmbeddedPiAgent(
28392849
messagingToolSentTexts: attempt.messagingToolSentTexts,
28402850
messagingToolSentMediaUrls: attempt.messagingToolSentMediaUrls,
28412851
messagingToolSentTargets: attempt.messagingToolSentTargets,
2852+
messagingToolSourceReplyPayloads: attempt.messagingToolSourceReplyPayloads,
28422853
heartbeatToolResponse: attempt.heartbeatToolResponse,
28432854
successfulCronAdds: attempt.successfulCronAdds,
28442855
};
@@ -2949,6 +2960,7 @@ export async function runEmbeddedPiAgent(
29492960
messagingToolSentTexts: attempt.messagingToolSentTexts,
29502961
messagingToolSentMediaUrls: attempt.messagingToolSentMediaUrls,
29512962
messagingToolSentTargets: attempt.messagingToolSentTargets,
2963+
messagingToolSourceReplyPayloads: attempt.messagingToolSourceReplyPayloads,
29522964
heartbeatToolResponse: attempt.heartbeatToolResponse,
29532965
successfulCronAdds: attempt.successfulCronAdds,
29542966
};
@@ -3064,6 +3076,7 @@ export async function runEmbeddedPiAgent(
30643076
messagingToolSentTexts: attempt.messagingToolSentTexts,
30653077
messagingToolSentMediaUrls: attempt.messagingToolSentMediaUrls,
30663078
messagingToolSentTargets: attempt.messagingToolSentTargets,
3079+
messagingToolSourceReplyPayloads: attempt.messagingToolSourceReplyPayloads,
30673080
heartbeatToolResponse: attempt.heartbeatToolResponse,
30683081
successfulCronAdds: attempt.successfulCronAdds,
30693082
};

0 commit comments

Comments
 (0)