feat(hooks): add before_llm_call plugin hook for LLM input interception#39195
feat(hooks): add before_llm_call plugin hook for LLM input interception#39195zeroaltitude wants to merge 1 commit intoopenclaw:mainfrom
Conversation
Adds the before_llm_call modifying hook that fires before every LLM API call within the agent loop. Plugins can inspect/modify the message context, filter the tool list, or block the call entirely. Key design decisions: - First-writer-wins for messages/systemPrompt (security plugins can't be overridden by lower-priority handlers) - Intersection latch for tools (a later handler can only narrow, never widen, the tool allowlist) - BeforeLlmCallBlockError sentinel class for graceful blocking without surfacing errors to the user - streamFn wrapper pattern (outermost wrapper, same context as agentLoop) Also moves clearInternalHooks() from server-startup.ts to server.impl.ts so plugin hooks registered during loadGatewayPlugins are preserved when bundled hooks load later. Files: - src/plugins/types.ts: BeforeLlmCallEvent/Result types - src/plugins/hooks.ts: runBeforeLlmCall runner - src/agents/pi-embedded-runner/run/hook-stream-wrapper.ts: streamFn wrapper - src/agents/pi-embedded-runner/run/attempt.ts: wrapper setup + iteration tracking - src/gateway/server-startup.ts, server.impl.ts: clearInternalHooks fix Tests: 7 stream wrapper + 8 merge semantics = 15 tests
Greptile SummaryThis PR introduces the The core implementation — the
Confidence Score: 3/5
|
| const { BeforeLlmCallBlockError } = await import("./hook-stream-wrapper.js"); | ||
| if (err instanceof BeforeLlmCallBlockError) { |
There was a problem hiding this comment.
Dynamic instanceof via catch-block import is fragile
BeforeLlmCallBlockError already carries the sentinel property isBeforeLlmCallBlock = true for exactly this situation. Using instanceof via a dynamic import in a catch block has a subtle failure mode: if a bundler or loader ever creates two separate module instances of hook-stream-wrapper.js (e.g., different chunks or a dual-package hazard), the class references won't be the same object and instanceof will return false. The block error would then fall through to the else branch, setting promptError and surfacing a run failure instead of the intended graceful termination.
The sentinel property is already there — prefer checking it directly:
| const { BeforeLlmCallBlockError } = await import("./hook-stream-wrapper.js"); | |
| if (err instanceof BeforeLlmCallBlockError) { | |
| const isBlockError = | |
| err instanceof Error && | |
| (err as { isBeforeLlmCallBlock?: boolean }).isBeforeLlmCallBlock === true; | |
| if (isBlockError) { |
This removes the dynamic import entirely and is immune to the dual-instance problem.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run/attempt.ts
Line: 1817-1818
Comment:
**Dynamic `instanceof` via catch-block import is fragile**
`BeforeLlmCallBlockError` already carries the sentinel property `isBeforeLlmCallBlock = true` for exactly this situation. Using `instanceof` via a dynamic import in a `catch` block has a subtle failure mode: if a bundler or loader ever creates two separate module instances of `hook-stream-wrapper.js` (e.g., different chunks or a dual-package hazard), the class references won't be the same object and `instanceof` will return `false`. The block error would then fall through to the `else` branch, setting `promptError` and surfacing a run failure instead of the intended graceful termination.
The sentinel property is already there — prefer checking it directly:
```suggestion
const isBlockError =
err instanceof Error &&
(err as { isBeforeLlmCallBlock?: boolean }).isBeforeLlmCallBlock === true;
if (isBlockError) {
```
This removes the dynamic import entirely and is immune to the dual-instance problem.
How can I resolve this? If you propose a fix, please make it concise.| const hookIterationUnsub = hookRunner?.hasHooks("before_llm_call") | ||
| ? activeSession.subscribe((event) => { | ||
| if (event.type === "turn_start") { | ||
| hookIterationRef.current++; | ||
| } | ||
| }) | ||
| : undefined; |
There was a problem hiding this comment.
Iteration counter is likely off-by-one for the first LLM call
hookIterationRef starts at 0, and the subscription increments it on every turn_start event. In the agent loop, turn_start fires before the LLM call (streamFn) in each turn. This means:
hookIterationRef.current = 0(initial)turn_startfires →hookIterationRef.current = 1- Wrapped
streamFnis called →event.iteration = 1← first call
The tests in hook-stream-wrapper.test.ts manually set iterationRef.current = 0 for the first call and = 1 for the second, establishing the expected contract that the first LLM call maps to iteration: 0. In production the first call will be iteration: 1, and subsequent calls will be iteration: N+1 relative to test expectations.
If turn_start does fire before streamFn, this is an off-by-one. Two straightforward fixes:
- Option A: Initialise
hookIterationRef = { current: -1 }so that after the firstturn_startincrement the first LLM call sees0. - Option B: Keep
{ current: 0 }and increment after the LLM call (e.g. onturn_endorllm_response).
It's worth verifying the turn_start / streamFn ordering in pi-agent-core to confirm the direction of the fix.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-runner/run/attempt.ts
Line: 1532-1538
Comment:
**Iteration counter is likely off-by-one for the first LLM call**
`hookIterationRef` starts at `0`, and the subscription increments it on every `turn_start` event. In the agent loop, `turn_start` fires **before** the LLM call (`streamFn`) in each turn. This means:
1. `hookIterationRef.current = 0` (initial)
2. `turn_start` fires → `hookIterationRef.current = 1`
3. Wrapped `streamFn` is called → `event.iteration = 1` ← first call
The tests in `hook-stream-wrapper.test.ts` manually set `iterationRef.current = 0` for the first call and `= 1` for the second, establishing the expected contract that the first LLM call maps to `iteration: 0`. In production the first call will be `iteration: 1`, and subsequent calls will be `iteration: N+1` relative to test expectations.
If `turn_start` does fire before `streamFn`, this is an off-by-one. Two straightforward fixes:
- **Option A:** Initialise `hookIterationRef = { current: -1 }` so that after the first `turn_start` increment the first LLM call sees `0`.
- **Option B:** Keep `{ current: 0 }` and increment *after* the LLM call (e.g. on `turn_end` or `llm_response`).
It's worth verifying the `turn_start` / `streamFn` ordering in `pi-agent-core` to confirm the direction of the fix.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fa757a11b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (err instanceof BeforeLlmCallBlockError) { | ||
| log.info(`LLM call blocked by before_llm_call hook: ${err.message}`); | ||
| } else { |
There was a problem hiding this comment.
Avoid reusing prior assistant output when hook blocks call
When before_llm_call blocks, this branch only logs and leaves promptError unset, so the attempt is treated as successful. In that path assistantTexts stays empty, and payload building falls back to lastAssistant from session history (src/agents/pi-embedded-runner/run/payloads.ts:195-251), which can resend a stale prior assistant reply if the conversation already had one. That turns a blocked call into incorrect user-visible output instead of a clean no-response outcome.
Useful? React with 👍 / 👎.
| "subagent_ended", | ||
| "gateway_start", | ||
| "gateway_stop", | ||
| "before_llm_call", |
There was a problem hiding this comment.
Enforce allowPromptInjection for before_llm_call hooks
This introduces before_llm_call as a fully valid typed hook, but it is not covered by the prompt-injection policy gate. registerTypedHook only enforces plugins.entries.<id>.hooks.allowPromptInjection=false for names in isPromptInjectionHookName (src/plugins/registry.ts:519), and that set still excludes before_llm_call (src/plugins/types.ts:380-383), so a plugin can still mutate messages/systemPrompt through this hook even when operators explicitly disable prompt injection for that plugin.
Useful? React with 👍 / 👎.
TL;DR
Adds the
before_llm_callmodifying plugin hook — fires before every LLM API call, allowing plugins to inspect/modify the message context, filter tools, or block the call.Split from #33916 (1 of 3). This PR covers input interception only. Tool-call gating (
after_llm_call) and output policies (before_response_emit) follow in stacked PRs.Hook contract
messagesAgentMessage[]systemPromptstringtoolsArray<{ name }>blockbooleanblockReasonstringArchitecture
Uses the
streamFnwrapper pattern — the hook runs insidestreamAssistantResponse()in theagentLoop's own async context. This is the outermost wrapper, so it sees the final context after all other transforms (cache trace, Ollama compat, xAI decode, Anthropic logging).BeforeLlmCallBlockErroris a sentinel class that propagates throughpi-agent-core's error handling and is caught gracefully inattempt.ts— the run ends cleanly with no assistant response rather than surfacing an error.Infrastructure fix
Moves
clearInternalHooks()fromserver-startup.tstoserver.impl.ts(before plugin registration). Without this, plugin hooks registered duringloadGatewayPluginswould be cleared when bundled hooks load later instartGatewaySidecars.Testing
pnpm checkclean (pre-existing streamSegments type mismatch excluded)AI-Assisted Development 🤖
Developed with AI assistance (Claude/OpenClaw agent "Tank"). All changes reviewed and directed by Eddie Abrams.