RFC: Expose existing InputProvenance / AgentInternalEvent signals to ContextEngine.assemble()

TL;DR. OpenClaw already has a complete InputProvenance model (src/sessions/input-provenance.ts) and an AgentInternalEvent taxonomy (src/agents/internal-event-contract.ts) that distinguishes third-party user messages, inter-session bridge messages, internal system events, and subagent announce / task-completion events. The runtime uses these internally in shouldSuppressAgentPromptPersistence() to decide what not to write to transcript. The same signals are absent from ContextEngine.assemble() params, so durable-store engines that rebuild context from their own DB cannot tell live volatile injections (subagent announces, inter-session bridges, internal-runtime-context blocks) apart from ordinary turns — and have to text-sniff for them. We propose exposing these signals via three optional, non-breaking assemble params. Zero changes to AgentMessage shape, zero new lifecycle hooks, zero changes to runtime persistence behavior. Engines that ignore the new fields behave identically to today.

1. Motivation

1.1 The contract gap

Current ContextEngine.assemble() (src/context-engine/types.ts):

assemble(params: {
  sessionId: string;
  sessionKey?: string;
  messages: AgentMessage[];
  tokenBudget?: number;
  availableTools?: Set<string>;
  citationsMode?: MemoryCitationsMode;
  model?: string;
  prompt?: string;        // added by #50848
}): Promise<AssembleResult>;

From inside assemble() the engine cannot reliably answer:

1. Where does pre-prompt history end and the live in-flight turn begin? (afterTurn gets prePromptMessageCount. assemble does not.)
2. Which messages were synthesized by the runtime as non-persistent live injections (subagent announces, retry/overflow re-prompts, internal-runtime-context blocks) vs ordinary durable user/assistant turns?
3. Which messages does the runtime know it will not write to transcript (i.e. shouldSuppressAgentPromptPersistence() returns true), so the engine should not ingest them?

1.2 The runtime already has all three signals

This RFC's central claim. We are not asking OpenClaw to invent new metadata. The signals already exist and are already used internally.

Signal A — InputProvenance (src/sessions/input-provenance.ts):

export const INPUT_PROVENANCE_KIND_VALUES = [
  "third-party_user",
  "inter_session",
  "internal_system",
] as const;

export type InputProvenance = {
  kind: InputProvenanceKind;
  originSessionId?: string;
  sourceSessionKey?: string;
  sourceChannel?: string;
  sourceTool?: string;  // e.g. "subagent_announce", "agent_step"
};

This is attached to user messages via applyInputProvenanceToUserMessage() (same file) and propagated through AgentCommandOpts.inputProvenance (src/agents/command/types.ts). At every assemble call site the runtime knows the provenance of the current turn's user message and can recover the provenance of prior turn messages from the transcript / session-context if needed.

Signal B — AgentInternalEvent (src/agents/internal-event-contract.ts, src/agents/internal-events.ts):

type AgentTaskCompletionInternalEvent = {
  type: typeof AGENT_INTERNAL_EVENT_TYPE_TASK_COMPLETION;
  source: AgentInternalEventSource;       // "subagent", "cron", etc.
  childSessionKey: string;
  // ... announce / task metadata
};

These travel on AgentCommandOpts.internalEvents (src/agents/command/types.ts) and are formatted into the prompt body by formatTaskCompletionEvent() / formatTaskCompletionEventForPlainPrompt(). The runtime knows which exact message in messages carries the formatted event because it just inserted it.

Signal C — prePromptMessageCount: already passed to afterTurn. Same number is available at the assemble call site; the runtime computes it once per attempt.

1.3 Existing internal consumer: shouldSuppressAgentPromptPersistence

The runtime's own decision logic for whether to persist a synthesized prompt to transcript reads exactly these signals (src/gateway/server-methods/agent.ts):

function shouldSuppressAgentPromptPersistence(params: {
  inputProvenance?: InputProvenance;
  internalEvents?: AgentInternalEvent[];
}): boolean {
  if (
    params.inputProvenance?.kind !== "inter_session" ||
    params.inputProvenance.sourceTool !== "subagent_announce"
  ) {
    return false;
  }
  return (
    params.internalEvents?.some(
      (event) =>
        event.type === AGENT_INTERNAL_EVENT_TYPE_TASK_COMPLETION && event.source === "subagent",
    ) === true
  );
}

This proves the signals are already structurally accessible at the call site. They're just not threaded into assemble.

1.4 Live consequence: lossless-claw Track 1 hotfix (#688)

Without these signals, Martian-Engineering/lossless-claw#688 (fix: preserve unpersisted volatile live input in LCM assembly, currently open) has to detect volatile-live-input tail messages by:

1. Walking params.messages from prePromptMessageCount forward (but prePromptMessageCount itself is not in params, so the engine computes a guess from its own DB state).
2. String-matching the user-message body for [Inter-session message], <<<BEGIN_OPENCLAW_INTERNAL_CONTEXT>>>, [Internal task completion event] markers.
3. Doing bipartite occurrence-matching against assembled DB output to avoid double-counting.
4. Evicting from the front of the assembled prefix if budget overflows.

The PR's own description acknowledges this is Track 1 (止血 / hot fix) and that the proper fix is a contract change. The marker strings come from src/sessions/input-provenance.ts (INTER_SESSION_PROMPT_PREFIX_BASE), src/agents/internal-runtime-context.ts (INTERNAL_RUNTIME_CONTEXT_BEGIN), and src/agents/internal-events.ts (formatTaskCompletionEvent) — i.e. the engine is doing inverse text-matching against the same OpenClaw code paths that emitted the messages, instead of receiving the structured metadata that drove the emission.

1.5 Other downstream cases

• Bug: sub-agent announce completion causes parent session to generate out-of-context replies #74286 ("Bug: sub-agent announce completion causes parent session to generate out-of-context replies"): the parent session's context engine reassembles from a durable store but the trigger message is a subagent announce. With provenance signals, an engine can detect a subagent-announce trigger and apply different inclusion logic (e.g. retain more recent turns).
• [Bug] TypeError at prompt assembly stage when lossless-claw is enabled (reading 'length' on undefined) #75541 ("TypeError at prompt assembly stage when lossless-claw is enabled"): downstream symptom of mismatched expectations between runtime and engine over what's in the assembled messages. Better contract signals reduce surface for this.
• Any third-party context engine that wants to do prefix-aligned RAG, identity-based anchoring, or live-input passthrough faces the same gap. lossless-claw is the example we have, not the only consumer this benefits.

1.6 What this RFC is not about

• Not adding fields to AgentMessage itself. The message.provenance extension already exists in applyInputProvenanceToUserMessage (object-spread, untyped on AgentMessage). This RFC does not formalize that — it threads parallel arrays through assemble params instead, which is a smaller surface than adding to the cross-package AgentMessage type.
• Not changing when assemble fires ([Feature]: ContextEngine.assemble should fire per LLM call, not per user prompt #73437's lane).
• Not changing how compaction is triggered (feat(context-engine): add interceptCompaction contract for context-engine plugins #81164's lane).
• Not asking the runtime to enforce persistence on the engine's behalf.

2. Proposal

Add three optional parameters to assemble. All optional, all backward-compatible.

assemble(params: {
  // existing fields unchanged

  /**
   * Number of messages that existed before the current prompt was issued.
   * `messages[0..prePromptMessageCount-1]` is pre-prompt history the engine
   * has (or should have) seen via prior afterTurn calls; the tail
   * `messages[prePromptMessageCount..]` is the current turn (prompt + any
   * runtime-injected entries that go with it).
   *
   * Same source of truth as the field already passed to `afterTurn`.
   * Optional; omitting it preserves today's behavior (engine cannot
   * distinguish pre-prompt vs live tail).
   */
  prePromptMessageCount?: number;

  /**
   * Per-message InputProvenance, aligned to `messages` by index.
   * Entries with no known provenance are `undefined` at that index.
   *
   * The runtime already attaches provenance to live user messages via
   * `applyInputProvenanceToUserMessage`. For ingested transcript messages
   * the runtime exposes whatever it can recover from `session-context` /
   * stored provenance, or `undefined`. Engines should treat `undefined`
   * as "no signal" rather than "third-party_user".
   *
   * When provided, `inputProvenance.length === messages.length`.
   */
  inputProvenance?: readonly (InputProvenance | undefined)[];

  /**
   * Per-message AgentInternalEvent annotation, aligned by index. An entry
   * at index `i` carries the internal event(s) that the runtime
   * synthesized into `messages[i]` (e.g. a subagent task-completion event
   * whose `formatTaskCompletionEvent()` output is the message body).
   * Entries with no associated internal event are `undefined`.
   *
   * Engines may use this to (a) detect that a tail user-role message is a
   * synthesized announce body rather than a real user input, (b) read the
   * structured event metadata (`childSessionKey`, `announceType`,
   * `status`, etc.) without parsing the formatted prompt body.
   *
   * When provided, `internalEvents.length === messages.length`.
   */
  internalEvents?: readonly (readonly AgentInternalEvent[] | undefined)[];
}): Promise<AssembleResult>;

2.1 Engine-side semantics (lossless-claw's planned use)

After this RFC lands, lossless-claw can replace its current text-marker sniff with a structured detection:

function isVolatileLiveInputAt(i: number, params: AssembleParams): boolean {
  // (a) Subagent / task-completion event marker
  if (params.internalEvents?.[i]?.some((e) =>
    e.type === AGENT_INTERNAL_EVENT_TYPE_TASK_COMPLETION
  )) return true;

  // (b) Inter-session bridge from a runtime tool
  if (params.inputProvenance?.[i]?.kind === "inter_session") return true;

  // (c) Tail position past pre-prompt boundary + no DB context_items row
  //     covering this identity → live volatile by elimination
  return false;
}

The new code is simpler than the current marker-based logic and is robust against the runtime renaming the formatted-event body or the inter-session prefix.

2.2 Runtime-side implementation sketch (~150 LOC excluding tests)

Single call site update on each runner:

1. src/agents/pi-embedded-runner/run/attempt.ts (runEmbeddedAttempt): compute the three arrays from already-available locals:

   • prePromptMessageCount — already computed for afterTurn.
   • inputProvenance[i] — for each message, read message.provenance if present (set by applyInputProvenanceToUserMessage); for prior turns, look up via the existing session-context provenance store if available, else undefined.
   • internalEvents[i] — when the runtime inserts the formatted body of an AgentInternalEvent into messages, it knows the index. Carry an Map<messageRef, AgentInternalEvent[]> from insertion site to the assemble call, project to the parallel array. (Sketch: instead of carrying a Map, attach to the message-ref hash; one of several easy implementations.)

2. extensions/codex/src/app-server/run-attempt.ts (codex harness): same. If prePromptMessageCount is harder to compute on the codex side, omit it (engine tolerates omission), keep the other two.

3. src/agents/internal-runtime-context.ts / src/agents/internal-events.ts: no change to existing exports; the runtime continues to format prompt bodies the same way. The new arrays are derived from sibling state, not from re-parsing message bodies.

4. Tests: extend src/agents/harness/context-engine-lifecycle.test.ts with new cases:

   • assemble receives prePromptMessageCount matching afterTurn.
   • assemble receives inputProvenance[i].kind === "inter_session" for an inter-session-tool-routed message.
   • assemble receives internalEvents[i] containing an AGENT_INTERNAL_EVENT_TYPE_TASK_COMPLETION for a subagent-announce body.
   • Backward compatibility: an engine ignoring the new fields gets identical messages.

2.3 Plugin SDK surface

• Add the three optional params to the assemble type in src/context-engine/types.ts.
• Re-export InputProvenance and AgentInternalEvent from the plugin-sdk barrel so engines can name the types without importing from internal paths. (InputProvenance is already in plugin-sdk/src/sessions/input-provenance.d.ts. AgentInternalEvent would need to be added.)
• Regenerate dist/plugin-sdk/src/context-engine/types.d.ts via the existing build pipeline.

2.4 What this RFC explicitly does not require

To stay narrow and avoid the rejection pattern of #80218 (RAG-in-assemble) / #77714 (pluggable memory backend) / #46234 (re-run sanitizers):

• No changes to AgentMessage shape itself.
• No new lifecycle hook ([Feature]: ContextEngine.assemble should fire per LLM call, not per user prompt #73437 owns "assemble per LLM call").
• No request for the engine to enforce persistence; the runtime stays the source of truth for transcript writes.
• No changes to compaction or intercept-compaction (feat(context-engine): add interceptCompaction contract for context-engine plugins #81164 owns that).
• No runtime-side text re-parsing; signals come from already-classified state.

3. Precedent

This proposal extends the same pattern that's been growing assemble's contract since #50848:

All four are non-breaking optional extensions. All follow "expose a signal the runtime already has, gate engine behavior on reading it." This RFC adds three more in the same shape.

4. Open questions

1. Naming. inputProvenance (matches existing InputProvenance type and AgentCommandOpts.inputProvenance) vs provenance (shorter at the call site). The current draft uses inputProvenance for symmetry; happy to flip.
2. Type access for engines. Re-exporting AgentInternalEvent from plugin-sdk exposes the internal event taxonomy to downstream consumers. That's intentional — it's how engines can use the structured event payload — but it does pin a more public contract for AgentInternalEvent. Alternative: expose only the discriminator (e.g. internalEventTypes?: readonly (readonly InternalEventType[] | undefined)[]) and keep the full event internal. Trade-off: less power for engines that want the structured fields, but smaller exposed surface.
3. Codex harness parity. PI-embedded runner has clean access to all three signals. Codex's extensions/codex/src/app-server/run-attempt.ts may have a thinner pipeline. Open to feedback on whether codex should pass through inputProvenance / internalEvents from day one or whether that's a follow-up.
4. Should we also pass an identityHashes array for prefix alignment? Speculative; the first three signals already resolve the immediate downstream pain. Happy to defer to a separate proposal if there's a stronger demand signal.

5. Migration / rollout

• Optional fields, zero behavior change for engines that don't consult them.
• lossless-claw will feature-detect the presence of the new fields. While both old and new OpenClaw versions are in production, the engine keeps its current text-marker sniff path as a fallback. Once the floor OpenClaw version has the signals for a release cycle, the marker constants and bipartite-matching logic are deleted.
• No deprecation needed in the runtime; the new fields are pure additions.

6. Why this isn't already covered by #73437 / #81079 / #81164

Each addresses a different dimension:

• [Feature]: ContextEngine.assemble should fire per LLM call, not per user prompt #73437 ("assemble per LLM call"): cadence — when does assemble fire. Orthogonal to what it gets.
• [feat]: thread currentTokenCount into ContextEngine.assemble #81079 ("currentTokenCount"): one specific token-accounting field for overflow guards.
• feat(context-engine): add interceptCompaction contract for context-engine plugins #81164 ("interceptCompaction"): engine participation in codex GPT-driven compaction. Different lifecycle.

This RFC sits on the dimension of "what context does assemble have about its inputs?", between #50848 (added the user prompt) and #81079 (adds the current token count).

7. Related issues / PRs

• Concurrent downstream hotfix: Martian-Engineering/lossless-claw#688 — text-marker-based volatile-input recovery. Track 1 stopgap; this RFC is Track 2 (the proper contract fix). lossless-claw#688 will be rewritten on top of this RFC's contract when both land.
• Direct motivating bug: Bug: sub-agent announce completion causes parent session to generate out-of-context replies #74286 (subagent announce → out-of-context parent reply).
• Related contract-extension precedent: feat(context-engine): pass incoming prompt to assemble #50848, fix(agents): filter runtime context from context engines #76251, [feat]: thread currentTokenCount into ContextEngine.assemble #81079, feat(context-engine): add interceptCompaction contract for context-engine plugins #81164.
• Adjacent: [Feature]: ContextEngine.assemble should fire per LLM call, not per user prompt #73437 (assemble cadence), [Bug] TypeError at prompt assembly stage when lossless-claw is enabled (reading 'length' on undefined) #75541 (downstream assembly TypeError), [Bug]: bootstrap/reconcile and hot-cache policy can leave deferred compaction debt stranded #67716 (bootstrap/reconcile + hot-cache deferred compaction).

8. Author notes

Filed by jetd1, maintainer of @martian-engineering/lossless-claw. We have shipped Track 1 (lossless-claw#688) to stop production bleeding while this RFC is discussed. Happy to drive the OpenClaw-side runtime patch as a follow-up PR (PI runner side at minimum; codex harness if a maintainer can confirm the wiring there).