fix(agents): unified fallback retry safety — images, partial execution, system context#44188
fix(agents): unified fallback retry safety — images, partial execution, system context#44188mitchmcalister wants to merge 2 commits intoopenclaw:mainfrom
Conversation
Greptile SummaryThis PR consolidates three draft PRs into a focused change that hardens the model fallback retry path across three dimensions: cross-provider image privacy, partial execution detection, and system context injection. Key changes:
Two minor style observations were flagged: (1) Confidence Score: 4/5
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84914158a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
240769d to
350de99
Compare
|
Fixed in 350de99 — gated The Cross-provider retries will now skip |
This comment was marked as spam.
This comment was marked as spam.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 350de99765
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| params.isFallbackRetry && | ||
| !!params.primaryProvider && | ||
| params.primaryProvider !== params.providerOverride; |
There was a problem hiding this comment.
Apply cross-provider image guard when primary is skipped
isCrossProviderRetry is gated on params.isFallbackRetry, but isFallbackRetry is derived from the number of executed runs, not the fallback candidate index. In runWithModelFallback, the primary candidate can be skipped entirely (for example, all profiles in cooldown), so the first executed cross-provider candidate is treated as non-retry and both resolveRetryImages(...) and suppressPromptImageDetection stay disabled, allowing local/attached images to be sent to a different provider despite the new privacy guard.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Verified the mechanism: fallbackAttemptIndex only increments when run() is called, so a skipped primary (cooldown) leaves isFallbackRetry = false on the first executed candidate.
However, this is the pre-existing behavior — before this PR, images always went to whatever provider ran first. This PR added protection for the retry case: images already sent to provider A should not be re-sent to provider B. The "primary skipped" scenario is a distinct design question about whether images should ever go cross-provider when the primary was never attempted — that is about the fallback trust model, not retry safety.
isFallbackRetry correctly reflects "a prior execution failed" vs "first attempt." A skip is not a failed execution — no data was sent to any provider.
Out of scope for this PR; worth tracking as a follow-up if the privacy posture requires it.
| partialExecution: (() => { | ||
| if (attempt.toolMetas.length === 0) { | ||
| return undefined; | ||
| } | ||
| const sanitizedToolNames = FailoverError.sanitizeToolNames( |
There was a problem hiding this comment.
Preserve partial execution across profile-rotation attempts
This partialExecution payload is built only from the current attempt.toolMetas, so tool activity from earlier in-run attempts is dropped when auth-profile rotation (advanceAuthProfile() + continue) occurred before final failover. In that case, if an earlier rotated attempt already executed side-effecting tools (especially messaging) and the last attempt failed before tools, the fallback model receives no partial-execution warning and can repeat actions that already ran.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Verified: runEmbeddedAttempt() creates a fresh subscription (and fresh toolMetas) on each loop iteration, so tools from an earlier profile-rotation attempt are indeed not carried into the final FailoverError.partialExecution.
This requires tool execution followed by an auth/rate-limit/timeout error within the same runEmbeddedPiAgent call — an edge case. The current behavior is strictly additive: we capture partial execution when the final attempt had tools, and when it did not (profile rotation reset it), the fallback model gets no warning, which is the same behavior as before this PR.
Accumulating toolMetas across profile rotations would require a loop-scoped accumulator. Valid enhancement, out of scope for this safety-focused PR.
|
The remaining |
…n, system context Rebased onto current main after upstream moved agent-command from src/commands/agent.ts to src/agents/agent-command.ts. - Cross-provider image privacy: strip images when fallback targets a different provider; suppress prompt-based image detection on cross-provider retry. - Partial execution detection: capture tool execution metadata from the embedded runner, attach sanitized tool names to FailoverError, and thread through the fallback loop. - System context injection: partial execution warnings injected via extraSystemPrompt; suppressed on cross-provider retries (CWE-200). - Prompt preservation: always re-send original prompt on fallback retry instead of generic "Continue where you left off" message. Closes openclaw#43481, closes openclaw#43482, closes openclaw#43492 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
350de99 to
0cff9a4
Compare
🔒 Aisle Security AnalysisWe found 2 potential security issue(s) in this PR:
Vulnerabilities1. 🔵 Unbounded tool name accumulation in fallback retries leaks internal tool names and bloats prompts
DescriptionIn
Vulnerable code: lastPartialExecution = lastPartialExecution
? {
...lastPartialExecution,
toolNames: [
...new Set([
...lastPartialExecution.toolNames,
...described.partialExecution.toolNames,
]),
],
didSendViaMessagingTool:
lastPartialExecution.didSendViaMessagingTool ||
described.partialExecution.didSendViaMessagingTool,
}
: described.partialExecution;RecommendationRe-apply sanitization/limits after merging tool names, so the cross-attempt accumulated state cannot grow beyond the intended cap. Example fix: import { FailoverError } from "./failover-error.js";
const merged = [
...new Set([
...lastPartialExecution.toolNames,
...described.partialExecution.toolNames,
]),
];
lastPartialExecution = {
...lastPartialExecution,
toolNames: FailoverError.sanitizeToolNames(merged),
didSendViaMessagingTool:
lastPartialExecution.didSendViaMessagingTool ||
described.partialExecution.didSendViaMessagingTool,
};Optionally, consider also enforcing a total character budget for 2. 🔵 Image sanitization bypass when suppressPromptImageDetection is enabled
DescriptionIn
Vulnerable code: const imageResult = params.suppressPromptImageDetection
? { images: params.images ?? [], detectedRefs: [], loadedCount: 0, skippedCount: 0 }
: await detectAndLoadPromptImages({ ... maxBytes, maxDimensionPx ... });
if (imageResult.images.length > 0) {
await abortable(activeSession.prompt(effectivePrompt, { images: imageResult.images }));
}RecommendationEnsure For example, keep suppression limited to detection/loading of prompt references, but still run sanitization on import { sanitizeImageBlocks } from "../../tool-images.js";
const imageSanitization = resolveImageSanitizationLimits(params.config);
const sanitizedExisting = params.images
? (await sanitizeImageBlocks(params.images, "prompt:images", {
maxBytes: MAX_IMAGE_BYTES,
maxDimensionPx: imageSanitization.maxDimensionPx,
})).images
: [];
const imageResult = params.suppressPromptImageDetection
? { images: sanitizedExisting, detectedRefs: [], loadedCount: 0, skippedCount: 0 }
: await detectAndLoadPromptImages({
...,
existingImages: params.images,
maxBytes: MAX_IMAGE_BYTES,
maxDimensionPx: imageSanitization.maxDimensionPx,
});Also consider adding an explicit invariant/guard that Analyzed PR: #44188 at commit Last updated on: 2026-03-25T01:57:39Z |
|
Rebased onto current main — the upstream file move (src/commands/agent.ts → src/agents/agent-command.ts) is resolved and CI should be clean. No upstream changes have addressed the root causes (prompt/image loss on fallback retry, cross-provider image disclosure, or partial execution blindness). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0cff9a4ddc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| lastError = isKnownFailover ? normalized : err; | ||
| const described = describeFailoverError(normalized); | ||
| previousFailureReason = described.reason ?? "unknown"; | ||
| lastPartialExecution = described.partialExecution ?? lastPartialExecution; |
There was a problem hiding this comment.
Merge partial-execution state across fallback failures
When multiple fallback attempts fail with partialExecution, runWithModelFallback overwrites earlier state with the latest one via lastPartialExecution = described.partialExecution ?? lastPartialExecution. This drops side-effect history from earlier failed attempts (for example, first attempt sent a message, second attempt only read a file), so later retries can lose the didSendViaMessagingTool warning and repeat user-visible actions. This should merge partial-execution state across attempts (union tool names and OR the messaging flag) rather than replace it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
fixed in 4cf0a10. lastPartialExecution now merges across attempts: union of tool names and OR of didSendViaMessagingTool, so side-effect history from earlier failed attempts carries forward. Added a test covering the multi-attempt merge case.
… partial execution state Address reviewer feedback on openclaw#44188: - Suppress detectAndLoadPromptImages on same-provider format-error retries, not just cross-provider retries. When images are intentionally stripped for a format failure, prompt-based image detection should also be suppressed to prevent re-loading local images (Aisle openclaw#2, CWE-200). - Merge partialExecution state across fallback attempts instead of replacing: union tool names and OR the didSendViaMessagingTool flag so side-effect history from earlier failed attempts carries forward (Codex review feedback). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Addressed in 4cf0a10: #1 (High): Acknowledged — other #2 (Medium): Fixed — #3 (Low): Intentional trade-off. Same-provider = same trust boundary; tool names are already sanitized via |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4cf0a10dd5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Fallback retries only fire on thrown errors (not partial streaming results), | ||
| // so the original prompt is always safe to re-send. The orphaned-user-message | ||
| // repair in attempt.ts handles any residual duplicate user turns. | ||
| const effectivePrompt = params.body; |
There was a problem hiding this comment.
Keep retry prompts generic for resumed CLI sessions
This change makes runAgentAttempt always resend params.body on fallback retries, but the “orphaned-user-message repair” only exists in the embedded runner path, not the CLI runner path. When a CLI provider retries against an existing cliSessionId (resume mode) after a timeout/rate-limit failure, the same user prompt can be appended again to that backend session, which can produce duplicated or inconsistent follow-up behavior. This regression is specific to resumed CLI sessions and was previously avoided by using the generic continuation retry prompt.
Useful? React with 👍 / 👎.
|
Re: Codex review on |
Summary
Consolidates three draft PRs (#43331, #43485, #43507) into a single focused change that handles fallback retry safety:
detectAndLoadPromptImages) on cross-provider retry to prevent local image files from being loaded and sent.toolMetas) from the embedded runner, attach sanitized tool names toFailoverError, and thread through the fallback loop withlastPartialExecution ?? previouspreservation for multi-hop chains.extraSystemPrompt(system-level context) rather than prepended to the user message. This keeps the warning in the proper prompt layer and avoids models treating[System: ...]markers as user text.Closes #43481, closes #43482, closes #43492
Architectural constraints
The pi-coding-agent library's
AgentSession.prompt()always adds a new user turn — there is no "retry last turn" API. The library's internal auto-retry usesagent.continue()which avoids duplicate turns, but this operates within a singlerunEmbeddedPiAgent()lifecycle. OpenClaw's model fallback system creates a new embedded runner invocation per candidate (new session setup, new streamFn), socontinue()cannot be used across fallback candidates without restructuring the runner lifecycle or modifying the library.The orphaned-user-message repair in
attempt.tscorrectly handles the re-submission pattern: it branches the session tree (pointer move, not deletion) to prevent consecutive user turns, then the original prompt is re-sent viaprompt(). This is working as designed.Alternatives explored
agent.continue()across fallback candidates: Not viable —continue()uses livestreamFnand model references, but the embedded runner configures these per-invocation. Would require sharing a session across candidates.sendCustomMessage({ deliverAs: "nextTurn" }): Public API for context injection, but queues for the nextprompt()call — timing doesn't align with the fallback retry flow.prompt()and reuses the orphaned user turn. Feasible but requires fundamental runner restructuring.Non-goals
Test plan
resolveRetryImagesunit tests: cross-provider strip, same-provider preserve, format error strip, undefined failureReasonresolveRetryImagesintegration test: images stripped through fullagentCommand→runEmbeddedPiAgentchainsuppressPromptImageDetectionintegration test: flag passed on cross-provider retryFailoverError.sanitizeToolNamesunit tests: character stripping, truncation, limit, empty filteringbuildPartialExecutionSystemContextunit tests: undefined/empty/with-tools/with-messaging-warningrunWithModelFallback: single-hop and multi-hop preservationpnpm checkcleanpnpm buildsucceeds