Skip to content

feat(hooks): add before_llm_call + after_llm_call plugin hooks [claude, human developer oversight]#39206

Closed
zeroaltitude wants to merge 30 commits intoopenclaw:mainfrom
zeroaltitude:feat/hook-after-llm-call
Closed

feat(hooks): add before_llm_call + after_llm_call plugin hooks [claude, human developer oversight]#39206
zeroaltitude wants to merge 30 commits intoopenclaw:mainfrom
zeroaltitude:feat/hook-after-llm-call

Conversation

@zeroaltitude
Copy link
Copy Markdown
Contributor

@zeroaltitude zeroaltitude commented Mar 7, 2026

TL;DR

Adds two sequential / mutating plugin hooks that allow policy plugins to intercept and gate LLM calls:

  • before_llm_call — inspect/modify messages, system prompt, filter tools, or block the call entirely
  • after_llm_call — block or filter tool calls produced by the LLM before any tool executes

No behavior changes unless a plugin registers these hooks.

New hooks (contracts)

Hook Fires Can modify Can block Primary use
before_llm_call Before each LLM API call messages / systemPrompt / tools yes Tool allowlisting, input policies, context injection
after_llm_call After LLM response, before tool execution toolCalls yes Batch tool-call auditing / filtering / blocking

These follow the same sequential "merge results across handlers" pattern as before_tool_call / message_sending.

Merge semantics

Field Semantics
messages First-writer-wins
systemPrompt First-writer-wins
tools Intersection latch — later handlers can only narrow, never widen
block One-way latch (once true, stays true)
blockReason First-writer-wins
toolCalls Intersection — later handlers can only narrow

Architecture

before_llm_call — streamFn wrapper

Uses the same streamFn wrapping pattern as cache-trace.ts and anthropic-payload-log.ts — outermost wrapper that sees the full context before delegating. The wrapper:

  1. Fires runBeforeLlmCall with the current context
  2. Applies modifications (message override, system prompt override, tool filtering)
  3. On block: true, throws BeforeLlmCallBlockError sentinel (caught gracefully in attempt.ts, not treated as a prompt failure)

after_llm_call — deterministic Promise-based gate

The after_llm_call hook needs to fire when the LLM response completes but enforce its decisions when tools execute — two different points in the agent loop. The gate bridge makes this deterministic:

agentLoop async context (single sequential flow):
  streamAssistantResponse()
    └── streamFn wrapper intercepts response completion
    └── extracts tool calls from final message
    └── fires runAfterLlmCall(), stores Promise in gate ← synchronous
    └── returns
  executeToolCalls()
    └── tool.execute()
        └── wrapToolWithBeforeToolCallHook
            └── await checkAfterLlmCallGate() ← first call blocks until hook resolves
                                                subsequent calls get cached result

Why this is deterministic: The Promise is guaranteed to exist before executeToolCalls() starts because both run in the same agentLoop async context, sequentially. streamAssistantResponse() must return before executeToolCalls() is called. The first tool call blocks until the hook resolves; subsequent calls in the same turn get the cached result instantly.

Fail-open on error: If an after_llm_call handler throws, the gate resolves to { blocked: false } — consistent with existing hook error handling posture.

Infrastructure fix

Moves clearInternalHooks() from server-startup.ts to server.impl.ts so plugin hooks registered during loadGatewayPlugins are not cleared when bundled hooks load later.

Review guide

  1. API surface: src/plugins/types.ts — new hook names, event types, result types
  2. Hook runner semantics: src/plugins/hooks.tsrunBeforeLlmCall + runAfterLlmCall (sequential/FIFO merge + error handling)
  3. before_llm_call integration: src/agents/pi-embedded-runner/run/hook-stream-wrapper.ts — streamFn wrapper, input interception + response interception
  4. after_llm_call gate: src/agents/pi-embedded-runner/run/after-llm-call-gate.ts — Promise-based gate bridge
  5. Gate enforcement: src/agents/pi-tools.before-tool-call.tsawait checkAfterLlmCallGate() before each tool
  6. Glue: src/agents/pi-embedded-runner/run/attempt.ts (hookCtx, iteration tracking, streamFn wrapping, cleanup), src/gateway/server.impl.ts, src/gateway/server-startup.ts

Security / correctness notes

  • before_llm_call blocking uses a sentinel error (BeforeLlmCallBlockError) that is caught gracefully — does not trigger prompt-error logging or retry logic
  • Tool intersection latch ensures later handlers can only narrow the tool set, never re-add removed tools
  • Gate clears on turn_start to prevent stale decisions from previous turns
  • Handler execution order is FIFO (registration order), consistent with registry expectations

Testing

  • Stream wrapper: 7 tests (before_llm_call interception, passthrough, error handling)
  • Gate bridge: 9 tests (block, filter, clear, isolation, async behavior, fail-open)
  • Hook runner merge semantics: 12 tests (intersection, first-writer-wins, one-way latch, no-handler paths)

28 total tests, pnpm check clean.

AI-Assisted Development 🤖

This PR was developed with AI assistance (Claude/OpenClaw agent "Tank").

  • Degree of testing: Comprehensive — 28 unit tests across 3 test files. pnpm check clean.
  • Human oversight: All changes reviewed and directed by Eddie Abrams
  • Understanding confirmed: The contributor understands the embedded agent loop, Promise-based gate architecture, streamFn wrapper chain, and hook merge semantics

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Mar 7, 2026

Greptile Summary

This PR adds two new sequential plugin hooks — before_llm_call and after_llm_call — enabling policy plugins to inspect, modify, or block LLM calls and their resulting tool executions from within the agent loop. The architecture is well-designed: the stream-wrapper pattern is consistent with existing wrappers, the Promise-based gate bridge is a clean solution to the ordering problem, and the prompt-injection constraint branch in registry.ts closes a real security gap. The clearInternalHooks() ordering fix in server.impl.ts / server-startup.ts is also correct.\n\nTwo issues need attention before merge:\n\n- before_dispatch duplicate in types.ts (P1 syntax): The diff accidentally inserted | \"before_dispatch\"; as part of the new additions before the original | \"before_dispatch\"; terminator, duplicating the entry in both the union type (where the second occurrence follows a ; that ends the declaration, risking a parse error) and in PLUGIN_HOOK_NAMES.\n\n- Gate cleanup key mismatch in attempt.ts (P1 logic): hook-stream-wrapper.ts stores gates under a composite sessionId:runId key, but both the turn_start clearing (line 2625) and the run-end cleanup (line 3203) call clearAfterLlmCallGate with just params.sessionId. This makes both cleanup sites no-ops when runId is defined — the intended memory-leak fix from the prior review round is effectively reverted.

Confidence Score: 3/5

Two targeted fixes needed before merge: the before_dispatch duplicate in types.ts and the gate cleanup key mismatch in attempt.ts.

Core architecture (stream wrapper, gate bridge, hook runner semantics, registry constraint) is solid and well-tested. Two concrete bugs need fixing: a copy-paste duplicate that may cause a TypeScript parse error in types.ts, and a cleanup key mismatch that re-introduces the memory leak the previous review flagged. Both are straightforward fixes but affect correctness.

src/plugins/types.ts (duplicate before_dispatch) and src/agents/pi-embedded-runner/run/attempt.ts (gate cleanup key at lines 2625 and 3203).

Important Files Changed

Filename Overview
src/plugins/types.ts Adds before_llm_call and after_llm_call hook types and handler map entries. Copy-paste error duplicates before_dispatch in both the union type and the PLUGIN_HOOK_NAMES array.
src/agents/pi-embedded-runner/run/attempt.ts Wires hookCtx, iterationRef, stream wrapper, and gate cleanup. Run-end and turn-level clearAfterLlmCallGate calls use sessionId alone, mismatching the composite sessionId:runId key used by the stream wrapper.
src/agents/pi-embedded-runner/run/after-llm-call-gate.ts New Promise-based gate module. Logic for set/check/clear is correct and key-agnostic. Fail-open on hook errors is intentional and consistent with the rest of the hook infrastructure.
src/agents/pi-embedded-runner/run/hook-stream-wrapper.ts New stream-wrapper implementing before_llm_call interception and after_llm_call gate-setting. Gate keying with composite sessionId:runId is correct here; sentinel BeforeLlmCallBlockError is handled cleanly.
src/agents/pi-tools.before-tool-call.ts Adds checkAfterLlmCallGate check before tool execution. Composite key construction matches the stream wrapper. Guard on hasHooks avoids unnecessary overhead when no hooks are registered.
src/plugins/hooks.ts Adds runBeforeLlmCall and runAfterLlmCall to the hook runner. Merge semantics (first-writer-wins, intersection latch, one-way block latch) are correct and well-tested.
src/plugins/registry.ts Adds prompt-injection constraint branch for before_llm_call. Correctly strips messages/systemPrompt from the return value while preserving block, blockReason, and tools for constrained plugins.
src/gateway/server.impl.ts Moves clearInternalHooks() earlier so plugin hooks registered during loadGatewayPlugins are not clobbered by bundled hook loading. Change is correct.
src/gateway/server-startup.ts Removes the misplaced clearInternalHooks() call (moved to server.impl.ts). Comment explains the ordering rationale clearly.

Comments Outside Diff (1)

  1. src/agents/pi-embedded-runner/run/attempt.ts, line 3198-3204 (link)

    Gate cleanup uses wrong key — memory leak remains

    The run-end cleanup calls clearAfterLlmCallGate(params.sessionId), but hook-stream-wrapper.ts stores the gate under a composite key (sessionId + ":" + runId) whenever runId is set — which it always is for normal embedded runs. Passing only params.sessionId here deletes a key that was never inserted, so the actual gate entry in the module-level gates Map is never removed. The same key mismatch exists in the turn_start subscription at line 2625.

    Both cleanup sites need to build the same composite key the wrapper does (see gateKey construction in hook-stream-wrapper.ts) and pass that to clearAfterLlmCallGate.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/plugins/types.ts
Line: 1424-1427

Comment:
**`before_dispatch` duplicated — copy-paste error**

The diff accidentally duplicated `"before_dispatch"` both in the union type and in `PLUGIN_HOOK_NAMES`. In the union, line 1426 terminates the type declaration with `;`, making line 1427 (`| "before_dispatch";`) an orphaned statement — this is a syntax error in TypeScript. In the array, lines 1457–1458 both contain `"before_dispatch"`, which causes `isPluginHookName` and the `pluginHookNameSet` to silently have a duplicate entry.

The correct fix is to only add the two new hooks before the existing `"before_dispatch"` terminator:

```suggestion
  | "before_llm_call"
  | "after_llm_call"
  | "before_dispatch";
```

And correspondingly in `PLUGIN_HOOK_NAMES` (lines 1455–1458), remove the duplicate `"before_dispatch"` on line 1458.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run/attempt.ts
Line: 3198-3204

Comment:
**Gate cleanup uses wrong key — memory leak remains**

The run-end cleanup calls `clearAfterLlmCallGate(params.sessionId)`, but `hook-stream-wrapper.ts` stores the gate under a composite key (`sessionId + ":" + runId`) whenever `runId` is set — which it always is for normal embedded runs. Passing only `params.sessionId` here deletes a key that was never inserted, so the actual gate entry in the module-level `gates` Map is never removed. The same key mismatch exists in the `turn_start` subscription at line 2625.

Both cleanup sites need to build the same composite key the wrapper does (see `gateKey` construction in `hook-stream-wrapper.ts`) and pass that to `clearAfterLlmCallGate`.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (8): Last reviewed commit: "Merge origin/main into feat/hook-after-l..." | Re-trigger Greptile

Comment thread src/agents/pi-embedded-runner/run/after-llm-call-gate.ts
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4b78cbd99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/plugins/types.ts Outdated
Comment thread src/agents/pi-embedded-runner/run/hook-stream-wrapper.ts Outdated
@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 831d18ddea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/pi-embedded-runner/run/hook-stream-wrapper.ts Outdated
@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

2 similar comments
@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e3a3181b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/pi-embedded-runner/run/attempt.ts
@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

2 similar comments
@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@greptile-apps Please rescore.

@zeroaltitude
Copy link
Copy Markdown
Contributor Author

Fixed in 652adeb. Removed the three phantom mock methods (runLoopIterationStart, runLoopIterationEnd, runBeforeResponseEmit) — they don't exist on HookRunner in this branch.

@zeroaltitude
Copy link
Copy Markdown
Contributor Author

@bmendonca3: Here is the updated branch from my split -- note the upgrade from best-effort to deterministically helpful for after-llm-call

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7a431896a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/pi-embedded-runner/run/after-llm-call-gate.ts
@zeroaltitude zeroaltitude force-pushed the feat/hook-after-llm-call branch from a7a4318 to 335124d Compare March 9, 2026 07:43
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 335124ddec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/plugins/hooks.ts Outdated
@zeroaltitude zeroaltitude force-pushed the feat/hook-after-llm-call branch from 335124d to a7ff574 Compare March 9, 2026 17:09
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 429fa933a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}
effectiveContext = {
...context,
systemPrompt: result.systemPrompt ?? context.systemPrompt,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve explicit empty systemPrompt overrides

before_llm_call treats systemPrompt: "" as a valid explicit override, but this merge uses nullish fallback (result.systemPrompt ?? context.systemPrompt), so an empty-string override is dropped and the previous system prompt is kept. That means policy plugins cannot reliably clear inherited prompt text, and stripped instructions can be unintentionally reintroduced on the model call.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already correct — the merge uses ?? (nullish coalescing), not ||: systemPrompt: result.systemPrompt ?? context.systemPrompt. An explicit "" override is preserved because ?? only falls back on null/undefined. Verified in tests.

// but session-level clearing must happen here on run end.
{
const { clearAfterLlmCallGate } = await import("./after-llm-call-gate.js");
clearAfterLlmCallGate(params.sessionId);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear after_llm_call gates with run-scoped keys

The gate is written with a run-scoped key (sessionId:runId) in the stream wrapper, but cleanup here deletes only params.sessionId. As a result, each run that sets an after_llm_call gate leaves its Promise in the module-level map, so long-lived gateways accumulate stale per-run entries and never reclaim them.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the integration merge — gates are keyed by ${sessionId}:${runId} and clearAfterLlmCallGate(sessionId) now clears all entries prefixed with that session. The turn_start subscription handler also clears via this key pattern. Verified — concurrent runs in interrupt mode don't cross-clear.

Comment on lines +218 to +221
arguments:
(part.arguments as Record<string, unknown>) ??
(part.input as Record<string, unknown>) ??
{},
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Clone tool-call arguments before after_llm_call handlers

This copies part.arguments/part.input by reference into event.toolCalls, and runModifyingHook passes that event object directly to handlers. If an after_llm_call plugin mutates event.toolCalls[i].arguments in place (for logging/normalization/etc.), it can rewrite the live tool payload that will be executed, even though this hook is supposed to only gate/block tool calls.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a42bc0dfireAfterLlmCallGate now passes structuredClone(finalMessage) to the hook instead of the live reference, preventing plugin handlers from mutating the LLM response mid-flight.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c80b6b565

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +3202 to +3203
const { clearAfterLlmCallGate } = await import("./after-llm-call-gate.js");
clearAfterLlmCallGate(params.sessionId);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear after_llm_call gate with the run-scoped key

The gate is written and read with a composite key (sessionId:runId), but this cleanup path deletes only params.sessionId. For normal embedded runs (which always have a run ID), the Promise entry is never removed from the module-level map, so long-lived gateways accumulate stale gate entries and can retain per-run decisions in memory indefinitely.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the integration merge — gates are keyed by ${sessionId}:${runId} and clearAfterLlmCallGate(sessionId) now clears all entries prefixed with that session. The turn_start subscription handler also clears via this key pattern. Verified — concurrent runs in interrupt mode don't cross-clear.

Comment on lines +218 to +220
arguments:
(part.arguments as Record<string, unknown>) ??
(part.input as Record<string, unknown>) ??
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clone tool-call arguments before after_llm_call handlers run

toolCalls[].arguments is populated from part.arguments/part.input by reference, and the hook runner passes that mutable event object directly into handlers. A handler that mutates event.toolCalls[i].arguments in place will rewrite the live tool payload that is executed later, even though after_llm_call is meant to be a block/filter gate rather than an argument-rewrite surface.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a42bc0dfireAfterLlmCallGate now passes structuredClone(finalMessage) to the hook instead of the live reference, preventing plugin handlers from mutating the LLM response mid-flight.

@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Mar 26, 2026
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f32298f3e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +107 to +111
effectiveContext = {
...context,
systemPrompt: result.systemPrompt ?? context.systemPrompt,
messages: (result.messages ?? context.messages) as Message[],
tools: filteredTools,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Emit llm_input after before_llm_call rewrites

before_llm_call now rewrites the actual request context here, but llm_input is still emitted earlier in runEmbeddedAttempt (before activeSession.prompt()), so hooks observing llm_input can receive unsanitized historyMessages/systemPrompt and can even log a call that is later blocked by BeforeLlmCallBlockError. In a policy stack where one plugin redacts sensitive content in before_llm_call and another audits llm_input, this leaks pre-redaction data and produces incorrect audit records; llm_input should be sourced from the post-hook effectiveContext (or emitted from this wrapper) so both hooks describe the same call.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid ordering concern — llm_input should see the post-before_llm_call-rewrite context, not the pre-rewrite one. Filed as a follow-up to reorder emissions so the llm_input observational hook reflects what actually gets sent to the LLM.

@openclaw-barnacle openclaw-barnacle Bot removed the docs Improvements or additions to documentation label Apr 2, 2026
Upstream moved hook types to hook-types.ts. Added before_llm_call and
after_llm_call types to new location. Adapted attempt.ts for upstream's
preflight recovery and yield abort patterns.
@openclaw-barnacle openclaw-barnacle Bot removed the gateway Gateway runtime label Apr 14, 2026
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91eb5c6b1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1335 to +1338
if (hookRunner?.hasHooks("before_llm_call") || hookRunner?.hasHooks("after_llm_call")) {
const { wrapStreamFnWithHooks } = await import("./hook-stream-wrapper.js");
activeSession.agent.streamFn = wrapStreamFnWithHooks(activeSession.agent.streamFn, {
hookRunner,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move hook stream wrapping after Google cache wrapper setup

Wrapping activeSession.agent.streamFn with wrapStreamFnWithHooks here makes the hook wrapper an inner layer once prepareGooglePromptCacheStreamFn is applied later in the same function (attempt.ts:1803-1819). The Google cache wrapper calls inner with systemPrompt removed (google-prompt-cache.ts:181-188 and :402-407), so on cache-eligible Google runs before_llm_call receives an empty system prompt and cannot reliably enforce/transform prompt policy against the real request context.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid layering concern — hook stream wrapping should compose after provider-specific wrappers (Google cache) so hooks see the provider's final output rather than intermediate chunks. Filed as a follow-up; requires careful sequencing to ensure hook block semantics still work with the cache wrapper's retry behaviour.

Prevents plugin handlers from mutating the live LLM response.
Uses structuredClone for a deep copy.
@chrismclarnon-ep
Copy link
Copy Markdown

Flagging an active downstream use-case for this PR that might be useful context.

I'm building a local in-process reliability gate stack that detects and blocks three failure modes we're seeing regularly in long-running agent sessions: tool-call simulation (narration that looks like a tool invocation without a real protocol tool_calls field), success claims not grounded in the turn's tool-execution ledger, and cron deliveries that fabricate output when required resources failed. Design doc and peer-review artefacts are in a local docs/design/reliability-layer/ directory in our workspace if useful.

Two of the three components (ledger-backed claim verification and fail-closed cron delivery) are fully buildable against the current runtime and don't depend on this PR. The third — self-correcting retry when an assistant emits tool-imitation text instead of a real tool call — requires the before_llm_call / after_llm_call hooks this PR introduces. Without the hooks we can detect and annotate; with them we can intercept and re-prompt.

Happy to help if it's useful:

  • Run the PR against our workload for a week and report hook-firing telemetry (we already have measurement-only baseline instrumentation live in prod capturing per-hour baseline ticks and per-turn detector fires)
  • Pressure-test with a corpus of ~15 adversarial cases we've seen in the wild
  • Contribute test vectors to the PR's existing test suite
  • Review the sequential-ordering and error-path handling in a specific commit if useful

Not trying to push a timeline — the current PR description notes [human developer oversight] on merge, which I respect. Just signalling that at least one real downstream workload exists that gates its upgrade path on this.

Also curious: are there known edge cases around hook ordering when multiple plugins register against the same hook point? I saw the Greptile summary reference sequential ordering — any guidance on how plugins should cooperate on inspecting or modifying the same pre-call state?

@aisle-research-bot
Copy link
Copy Markdown

aisle-research-bot Bot commented Apr 22, 2026

🔒 Aisle Security Analysis

We found 5 potential security issue(s) in this PR:

# Severity Title
1 🟠 High Fail-open hook error handling allows bypass of LLM/tool policy enforcement
2 🟠 High Prompt-injection policy bypass in before_llm_call due to incorrect hook classification and mutable shared event
3 🟠 High after_llm_call hook can tamper tool-call arguments via shared object references
4 🟡 Medium Unhandled dynamic import in finally can abort cleanup and mask original errors
5 🟡 Medium after_llm_call gate Map leak due to inconsistent keying (sessionId vs sessionId:runId)
1. 🟠 Fail-open hook error handling allows bypass of LLM/tool policy enforcement
Property Value
Severity High
CWE CWE-284
Location src/agents/pi-embedded-runner/run/after-llm-call-gate.ts:60-86

Description

The new LLM hook enforcement path fails open when hook execution errors occur, which can bypass security/policy plugins if they are intended as mandatory controls.

after_llm_call gate (tool execution)

  • setAfterLlmCallGatePromise() converts a rejected after_llm_call hook promise into { blocked: false }.
  • This means any tool calls in that turn will be allowed if the hook fails (e.g., network/policy service outage, thrown exception).

before_llm_call (prompt/tool filtering)

  • wrapStreamFnWithHooks() catches errors from runBeforeLlmCall() and continues with the original context/tools.
  • If a security plugin is responsible for filtering tools or blocking unsafe prompts, an attacker may be able to trigger an error condition (e.g., by causing the plugin to throw on malformed input or by inducing transient dependency failures) to bypass the protections.

Vulnerable code (after_llm_call gate):

.catch((err) => {
  log.warn(`after_llm_call hook error (failing open): ${String(err)}`);
  return { blocked: false } as GateDecision;
});

Vulnerable code (before_llm_call):

} catch (err) {// Fail-open: log and continue with the original context.
  log.warn(`before_llm_call hook failed: ${String(err)}`);
}

Recommendation

Treat policy hooks as fail-closed by default, or make the behavior explicitly configurable per hook (mandatory vs best-effort).

Minimum fix (fail-closed for after_llm_call):

.catch((err) => {
  log.error(`after_llm_call hook error (failing closed): ${String(err)}`);
  return { blocked: true, blockReason: "after_llm_call hook failed" } as GateDecision;
});

For before_llm_call, either:

  • fail closed by throwing a block error when the hook fails, or
  • continue but restrict capabilities (e.g., remove all tools) until policy can be evaluated.

Also consider:

  • adding timeouts and circuit-breaking;
  • emitting structured audit events when enforcement fails;
  • documenting that hooks used for security controls must not fail open in production.
2. 🟠 Prompt-injection policy bypass in before_llm_call due to incorrect hook classification and mutable shared event
Property Value
Severity High
CWE CWE-693
Location src/plugins/registry.ts:1193-1232

Description

The allowPromptInjection=false policy is intended to constrain prompt-mutating hooks, but the new before_llm_call hook is not treated as a prompt-injection hook by isPromptInjectionHookName(). As a result, the constraint wrapper for before_llm_call is never applied, and a plugin configured with allowPromptInjection=false can still freely mutate event.messages/event.systemPrompt.

Additionally, even in the (intended) wrapper, only messages is deep-cloned; other fields (notably tools) are passed by reference via ...event, so a handler can mutate event.tools in-place to influence subsequent handlers and downstream tool filtering decisions.

Key points:

  • Policy gate never triggers for before_llm_call because PROMPT_INJECTION_HOOK_NAMES excludes it.
  • Hook runner executes handlers sequentially with the same event object, so in-place mutations persist across handlers.

Vulnerable code (policy bypass and partial cloning):

if (policy?.allowPromptInjection === false && isPromptInjectionHookName(hookName)) {// ...
  if (hookName === "before_llm_call") {
    const sandboxedEvent: PluginHookBeforeLlmCallEvent = {
      ...event,
      messages: structuredClone(event.messages),
      systemPrompt: event.systemPrompt,
    };
    const result = await original(sandboxedEvent, ctx);
    const { messages: _m, systemPrompt: _s, ...safe } = result;
    return safe;
  }
}

And the prompt-injection hook allowlist omits before_llm_call:

export const PROMPT_INJECTION_HOOK_NAMES = [
  "before_prompt_build",
  "before_agent_start",
] as const;

Impact:

  • A plugin believed to be “non prompt-injecting” (allowPromptInjection=false) can still modify the LLM input (messages/systemPrompt) via before_llm_call.
  • In-place mutation of tools can affect later handlers’ view of the available tools and potentially weaken tool allowlisting/auditing logic that assumes events are immutable per handler.

Recommendation

  1. Ensure before_llm_call is classified as a prompt-injection hook when allowPromptInjection=false, or remove the isPromptInjectionHookName() guard and explicitly check each constrained hook.

  2. Prevent cross-handler interference by deep-cloning (or freezing) all mutable fields exposed to handlers (at least tools, and ideally the whole event), e.g.:

// hook-types.ts
export const PROMPT_INJECTION_HOOK_NAMES = [
  "before_prompt_build",
  "before_agent_start",
  "before_llm_call",
] as const;// registry.ts wrapper
const sandboxedEvent: PluginHookBeforeLlmCallEvent = structuredClone(event);// or at minimum:
const sandboxedEvent: PluginHookBeforeLlmCallEvent = {
  ...event,
  messages: structuredClone(event.messages),
  tools: structuredClone(event.tools),
};
Object.freeze(sandboxedEvent.tools);

This ensures plugins cannot bypass the configured policy and cannot mutate shared objects to influence other handlers.

3. 🟠 after_llm_call hook can tamper tool-call arguments via shared object references
Property Value
Severity High
CWE CWE-471
Location src/agents/pi-embedded-runner/run/hook-stream-wrapper.ts:208-222

Description

In fireAfterLlmCallGate() the toolCalls array is built by directly reusing the arguments/input object from finalMessage.content without cloning.

  • finalMessage is the same assistant message object that will later be used by the agent runtime when executing tool calls.
  • after_llm_call handlers receive event.toolCalls[i].arguments and can mutate it (including nested fields).
  • Because these arguments objects are shared references to finalMessage.content[*].arguments/input, such mutations can alter the tool-call payload that will subsequently be executed.

This enables plugin-driven tool argument tampering (confused-deputy) even though the after-LLM hook/gate is intended to only filter/block tool calls.

Vulnerable code:

arguments:
  (part.arguments as Record<string, unknown>) ??
  (part.input as Record<string, unknown>) ??
  {},

Recommendation

Deep-clone tool-call argument objects before exposing them to plugins (and ideally freeze/sandbox the event) so plugin handlers cannot mutate the assistant message that will be executed.

Example fix:

const rawArgs =
  (part.arguments as Record<string, unknown>) ??
  (part.input as Record<string, unknown>) ??
  {};

toolCalls.push({
  id: String(part.id ?? ""),
  name: String(part.name ?? ""),
  arguments: structuredClone(rawArgs),
});

Additionally consider cloning the entire toolCalls array passed to runAfterLlmCall (or freezing it) and/or ensure tool execution uses a separately-parsed immutable copy of tool calls rather than the mutable finalMessage object.

4. 🟡 Unhandled dynamic import in finally can abort cleanup and mask original errors
Property Value
Severity Medium
CWE CWE-755
Location src/agents/pi-embedded-runner/run/attempt.ts:2594-2599

Description

In runEmbeddedAttempt, the finally cleanup block performs an awaited dynamic import and gate cleanup without a protective try/catch.

  • If await import("./after-llm-call-gate.js") fails (packaging/bundling issues, missing file, transient loader errors) or clearAfterLlmCallGate(...) throws, the exception will be thrown from the finally block.
  • Exceptions thrown from finally can mask the original error from the main try block.
  • More importantly, throwing here can prevent subsequent teardown steps from running (e.g., detachBackend, clearActiveEmbeddedRun, abort listener removal), potentially causing resource leaks and unbounded growth / stuck sessions.

Vulnerable code:

{
  const { clearAfterLlmCallGate } = await import("./after-llm-call-gate.js");
  clearAfterLlmCallGate(params.sessionId);
}

Recommendation

Wrap the dynamic import and gate cleanup in a try/catch inside finally, and ensure later teardown steps always run.

try {
  const mod = await import("./after-llm-call-gate.js");
  mod.clearAfterLlmCallGate(params.sessionId);
} catch (err) {
  log.error(
    `CRITICAL: failed to clear after_llm_call gate: runId=${params.runId} sessionId=${params.sessionId} ${String(err)}`,
  );
}

Optionally, avoid dynamic import in teardown by importing the module statically at top-level if feasible in your build, or guard the call when params.sessionId is falsy.

5. 🟡 after_llm_call gate Map leak due to inconsistent keying (sessionId vs sessionId:runId)
Property Value
Severity Medium
CWE CWE-400
Location src/agents/pi-embedded-runner/run/attempt.ts:1720-1729

Description

after_llm_call gate entries are stored under a per-run key (sessionId:runId) in the stream wrapper and are read under the same key in the tool wrapper, but runEmbeddedAttempt() clears the gate using only sessionId.

  • Write key (hook stream wrapper): gateKey = ${sessionId}:${runId}`` when runId is present
  • Read key (before-tool-call): same sessionId:runId key
  • Clear key (attempt.ts turn_start + finally): clearAfterLlmCallGate(params.sessionId)

Impact:

  • When params.runId is set (normal for runs), the module-level gates Map in after-llm-call-gate.ts never deletes the sessionId:runId entry.
  • Over time, repeated runs/sessions accumulate in the Map, causing unbounded memory growth and potential process OOM / DoS.

Vulnerable code (cleanup uses wrong key):

// attempt.ts
clearAfterLlmCallGate(params.sessionId);

Recommendation

Ensure clearAfterLlmCallGate() is called with the same key used by writers/readers (i.e., include runId when present). Ideally centralize key construction in a helper to avoid future drift.

Example fix:

// attempt.ts
const gateKey = params.runId ? `${params.sessionId}:${params.runId}` : params.sessionId;
clearAfterLlmCallGate(gateKey);

Also apply the same gateKey derivation in both the turn_start subscription and the finally cleanup block.


Analyzed PR: #39206 at commit 0f01099

Last updated on: 2026-04-23T05:18:40Z

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf73e64910

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +218 to +220
arguments:
(part.arguments as Record<string, unknown>) ??
(part.input as Record<string, unknown>) ??
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Clone tool-call arguments before running after_llm_call

Fresh evidence in this commit: toolCalls[].arguments is taken directly from part.arguments/part.input, and only response is cloned later, so event.toolCalls still points at live payload objects. If an after_llm_call handler mutates event.toolCalls[i].arguments (for normalization, redaction, etc.), it can silently change the tool input that gets executed, even though this hook is documented as a block/filter gate rather than an argument-rewrite surface.

Useful? React with 👍 / 👎.

Upstream now declares 'let lastAssistant' earlier in the function.
Our block-response override was redeclaring with 'const', causing
TS2451. Use reassignment via if/else instead.
@vincentkoc vincentkoc self-assigned this Apr 25, 2026
@vincentkoc
Copy link
Copy Markdown
Member

Closing this as superseded by the current hook architecture on main.

The exact before_llm_call / after_llm_call mutating hook names from this branch did not land as-is, but the accepted surfaces now cover the supported version of this capability:

  • before_model_resolve / before_prompt_build for pre-call model and prompt policy
  • before_agent_reply for short-circuiting a turn before the model path continues
  • before_tool_call for blocking, approving, or rewriting tool execution before any tool runs
  • model_call_started / model_call_ended plus llm_input / llm_output for model-call observation and diagnostics

The remaining part of this PR is a stale XL branch with failing extension checks, and keeping it open would point contributors at the wrong API shape. Thanks @zeroaltitude for pushing the original design forward; the useful pieces have been absorbed into the newer split hook model.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants