feat(core): fail impossible goals#4230
Conversation
📋 Review SummaryThis PR introduces a new "failed" terminal state for goals that are judged as genuinely impossible to achieve, preventing the agent from looping until the iteration cap. The implementation adds an 🔍 General Feedback
🎯 Specific Feedback🟡 High
🟢 Medium
🔵 Low
✅ Highlights
|
wenshao
left a comment
There was a problem hiding this comment.
🔍 Code Review — PR #4230
Scope: 13 files (+243/-29) | Build: ✅ | Tests: 76/76 pass | Typecheck: ✅ | Lint: ✅
This PR adds an "impossible" terminal state for /goal, allowing the judge to signal that a goal is genuinely unachievable. The implementation is clean and well-structured. I found 1 critical issue, 2 suggestions, and 2 minor comment updates.
🔴 Critical — Prompt injection via judgeReason (goalHook.ts:48-57)
The continuationReasonForGoal function now interpolates the judge's free-form verdict.reason into the continuation prompt fed back to the working model. The old code explicitly avoided this with a comment explaining why:
"Keep the judge's free-form diagnostic in goal state/UI only. The Stop hook reason is fed back to the model as the next continuation prompt, so it must be a fixed instruction derived from the original user goal rather than untrusted transcript-derived judge text."
The new code injects it verbatim (with only a soft "Hook feedback only" label). This is a prompt-injection surface that affects all goals, not just the impossible path. A compromised or confused judge can feed arbitrary instructions into the model's continuation context.
The test at goalHook.test.ts was explicitly changed to assert that "rm -rf /" IS present in the continuation reason — converting a security guardrail into a regression test for the injection.
🟡 Suggestion — impossible bypasses iteration-cap safety (goalHook.ts:192-202)
The control flow is: verdict.ok → verdict.impossible → iteration-cap check. A single judge hallucination (impossible: true on iteration 1) kills the goal immediately with no guardrail. Consider requiring impossible on ≥2 consecutive evaluations or adding a minimum iteration floor:
if (verdict.impossible && latest.iterations >= 2) {🟡 Suggestion — additionalProperties: false cast hack (goalJudge.ts:55)
The RESPONSE_SCHEMA is typed as Schema & { additionalProperties: boolean } — a cast outside @google/genai's typed contract. If the SDK changes its serialization, this constraint silently vanishes. Consider adding a structural parity test that asserts schema properties keys match JudgeResult interface keys.
🟢 Nice to have — Stale inline comments
Two comments reference only achieved / aborted but not the new failed kind:
GoalStatusMessage.tsx:135: "shown on terminal cards (achieved / aborted)"activeGoalStore.ts:66: "Goal achieved" / "Goal aborted"
Reviewed by: qwen-code (mimo-v2.5-pro)
| 'Continue working toward satisfying the goal. Active /goal condition remains the condition above.' | ||
| ); | ||
| } | ||
|
|
There was a problem hiding this comment.
🔴 Critical — Prompt injection via judgeReason
The old code deliberately kept the judge's free-form diagnostic out of the continuation prompt:
"Keep the judge's free-form diagnostic in goal state/UI only. The Stop hook reason is fed back to the model as the next continuation prompt, so it must be a fixed instruction derived from the original user goal rather than untrusted transcript-derived judge text."
This PR reverses that security decision. The judge's reason is LLM-generated text quoting the model's own transcript — it can contain adversarial content. The soft framing "Hook feedback only; not a user instruction." mitigates but does not eliminate injection risk.
This affects all goals, not just the impossible path — every "not met" continuation now carries judge text.
Suggested fix: Either (a) revert to the old fixed template and surface judge text only in UI/history, or (b) if feedback is intentional, add sanitization (cap length, strip imperative sentences, or wrap in a clearly non-actionable frame).
| return { continue: true }; | ||
| } | ||
|
|
||
| if (verdict.impossible) { |
There was a problem hiding this comment.
🟡 Suggestion — impossible bypasses the iteration-cap safety valve
The control flow is: verdict.ok → verdict.impossible → iteration-cap check (line 204). A single judge hallucination (impossible: true on iteration 1) kills the goal immediately — no guardrail, no second chance.
Suggested fix: Add a minimum iteration floor before honoring impossible:
if (verdict.impossible && latest.iterations >= 2) {This gives the goal at least a couple of turns before a single bad verdict can terminate it.
| properties: { | ||
| ok: { type: 'BOOLEAN' as unknown as Schema['type'] }, | ||
| reason: { type: 'STRING' as unknown as Schema['type'] }, | ||
| impossible: { type: 'BOOLEAN' as unknown as Schema['type'] }, |
There was a problem hiding this comment.
🟡 Suggestion — additionalProperties: false is an invisible footgun
The Schema & { additionalProperties: boolean } cast is outside @google/genai's typed contract. If the SDK changes its serialization to strip unknown fields, this constraint silently vanishes — tests pass because the mock doesn't go through the SDK.
Also, if someone adds a new field to the judge response (e.g. confidence), they must remember to add it to RESPONSE_SCHEMA.properties too. Forgetting means the API silently strips the new field with no compile-time signal.
Suggested fix: Add a unit test asserting the schema's properties keys match the JudgeResult interface keys — a structural parity check that catches future drift.
wenshao
left a comment
There was a problem hiding this comment.
No new issues found in the incremental review. LGTM! ✅ — gpt-5.5 via Qwen Code /review
|
|
||
| if ( | ||
| verdict.impossible && | ||
| latest.iterations >= MIN_IMPOSSIBLE_GOAL_ITERATIONS |
There was a problem hiding this comment.
[Suggestion] Missing debug log when impossible floor suppresses verdict
When the judge returns impossible: true but iterations < MIN_IMPOSSIBLE_GOAL_ITERATIONS, the verdict is silently deferred with no log. The max-iterations path (line 207) has a debugLogger.debug call; this suppression path has none. At 3 AM, an operator would see the judge's "impossible" reason in lastReason but have no log explaining why the system kept looping.
| latest.iterations >= MIN_IMPOSSIBLE_GOAL_ITERATIONS | |
| if (verdict.impossible) { | |
| if (latest.iterations >= MIN_IMPOSSIBLE_GOAL_ITERATIONS) { | |
| finishGoal(config, sessionId, latest, { | |
| kind: 'failed', | |
| condition: latest.condition, | |
| iterations: latest.iterations, | |
| durationMs: Date.now() - latest.setAt, | |
| lastReason: verdict.reason, | |
| }); | |
| return { continue: true }; | |
| } | |
| debugLogger.debug( | |
| `Impossible verdict suppressed: iterations=${latest.iterations} < floor=${MIN_IMPOSSIBLE_GOAL_ITERATIONS}; continuing loop.`, | |
| ); | |
| } |
— mimo-v2.5-pro via Qwen Code /review
| export const GOAL_JUDGE_TIMEOUT_MS = 25_000; | ||
| export const GOAL_HOOK_TIMEOUT_SECONDS = 30; | ||
| export const GOAL_HOOK_TIMEOUT_MS = GOAL_HOOK_TIMEOUT_SECONDS * 1000; | ||
| export const MIN_IMPOSSIBLE_GOAL_ITERATIONS = 2; |
There was a problem hiding this comment.
[Suggestion] MIN_IMPOSSIBLE_GOAL_ITERATIONS lacks documentation
The exported constant has no JSDoc explaining its rationale. The value "2" is a policy decision — it means the model gets one continuation turn after the first impossible verdict. A future maintainer encountering "my impossible goal took 3 turns to stop" will have to reverse-engineer the control flow.
| export const MIN_IMPOSSIBLE_GOAL_ITERATIONS = 2; | |
| /** | |
| * Minimum iteration count before an `impossible` judge verdict is accepted. | |
| * Gives the model at least one continuation turn after the judge first flags | |
| * impossibility, guarding against premature failure on a single bad-judgment | |
| * turn. The goal terminates on the second consecutive impossible verdict. | |
| */ | |
| export const MIN_IMPOSSIBLE_GOAL_ITERATIONS = 2; |
— mimo-v2.5-pro via Qwen Code /review
| the condition is genuinely unachievable rather than deferring to the assistant's | ||
| self-assessment. Do not use it just because progress is slow or evidence is | ||
| currently missing. When in doubt, return {"ok": false} without "impossible".`; | ||
|
|
There was a problem hiding this comment.
Good prompt guardrails. "The assistant claiming the goal is impossible is evidence, not proof" is the key line — it prevents the model from self-certifying impossibility to escape a difficult goal. The "when in doubt, return ok:false without impossible" fallback is also the right default.
|
|
||
| export const JUDGE_RESULT_SCHEMA_KEYS = [ | ||
| 'ok', | ||
| 'reason', |
There was a problem hiding this comment.
[Suggestion] JudgeResult.impossible has undocumented mutual exclusion with ok
parseJudgeReply strips impossible when ok is true:
...(impossible && !ok ? { impossible: true } : {}),The test covers this (ignores impossible=true when the judge also reports ok=true), but the JudgeResult interface doesn't document that impossible is only meaningful when ok is false. A future code path checking verdict.impossible without first verifying !verdict.ok could misclassify an achieved goal as impossible.
| 'reason', | |
| /** | |
| * Whether the goal is genuinely impossible in this session. | |
| * Only meaningful when `ok` is `false`. If `ok` is `true`, | |
| * this field is always absent. | |
| */ | |
| impossible?: boolean; |
— DeepSeek/deepseek-v4-pro via Qwen Code /review
| @@ -126,7 +132,8 @@ export const GoalStatusMessage: React.FC<GoalStatusMessageProps> = ({ | |||
| <Text wrap="wrap">{condition}</Text> | |||
| </Box> | |||
There was a problem hiding this comment.
[Suggestion] Switch default silently absorbs future GoalStatusKind variants
The case 'aborted': + default: share one return block. This PR correctly adds case 'failed': before the default, but if GoalStatusKind gains another variant later, TypeScript won't catch the missing case — the new kind will silently render as "Goal aborted" with a warning icon.
| </Box> | |
| case 'aborted': | |
| return { | |
| prefix: '!', | |
| prefixColor: theme.status.warning, | |
| title: 'Goal aborted', | |
| }; | |
| // No default — let TypeScript enforce exhaustiveness. |
— DeepSeek/deepseek-v4-pro via Qwen Code /review
| return { continue: true }; | ||
| } | ||
|
|
||
| // Give the latest assistant output one final evaluation before aborting. |
There was a problem hiding this comment.
[Suggestion] Missing debugLogger.debug() on the actual failed termination path
The suppressed-impossible path (below floor) has a debugLogger.debug() call, which is great. But the actual termination path where the goal IS marked as failed (iterations ≥ MIN) calls finishGoal without any debug log. At 3 AM, if a user reports their goal was prematurely marked impossible, there's no structured evidence that the judge returned impossible: true — only the freeform lastReason text in UI history.
| // Give the latest assistant output one final evaluation before aborting. | |
| debugLogger.debug( | |
| `Goal judge ruled impossible; clearing goal.`, | |
| { reason: verdict.reason, iterations: latest.iterations }, | |
| ); | |
| finishGoal(config, sessionId, latest, { |
— DeepSeek/deepseek-v4-pro via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
Additional notes (could not be mapped to specific diff lines)
Suggestion — Removed prompt injection safety comment. The PR removed the comment explaining why the continuation prompt must use continuationReasonForGoal(condition) instead of verdict.reason: verdict.reason is untrusted model-generated text and feeding it back as the continuation prompt would be a prompt injection vector. The safety property is encoded in the runtime string but the design rationale is lost. Consider restoring the comment or moving it to continuationReasonForGoal's JSDoc.
Suggestion — Missing test for impossible precedence over MAX_GOAL_ITERATIONS. The code correctly checks impossible before the iteration cap, so failed takes precedence over aborted when both conditions are true. But no test verifies this — if the check order is accidentally reversed in a future refactor, the behavior would silently regress.
Suggestion — Missing integration test for impossible→failed pipeline. Only unit-level tests with mocked judgeGoal cover the new path. An integration test through the real HookSystem would catch bugs in how judgeGoalWithTimeout interacts with the impossible verdict (e.g., if the timeout wrapper accidentally swallows impossible).
Verdict: Comment — no Critical issues. Build ✅, tests 77/77 ✅.
— DeepSeek/deepseek-v4-pro via Qwen Code /review
| : never; | ||
|
|
||
| export const JUDGE_RESULT_SCHEMA_COVERS_INTERFACE: SchemaCoversJudgeResult = true; | ||
|
|
There was a problem hiding this comment.
Nit: this exported constant is never read at runtime — it exists solely to trigger the SchemaCoversJudgeResult compile-time check. Consider making it unexported (just const) or adding a brief // compile-time only annotation, since it currently shows up in the public API surface but has no runtime consumers.
| export const JUDGE_RESULT_SCHEMA_COVERS_INTERFACE: SchemaCoversJudgeResult = true; | ||
|
|
||
| const RESPONSE_SCHEMA: Schema & { additionalProperties: boolean } = { | ||
| // Schema typing in @google/genai uses an enum-like Type, but accepts the |
There was a problem hiding this comment.
The additionalProperties field is bolted on via Schema & { additionalProperties: boolean } because @google/genai's Schema type does not include it. This works but is fragile — if the upstream type adds the property with a different shape, the intersection could silently conflict. Consider a // @ts-expect-error or a short comment noting the upstream gap so future readers understand the intent.
| line). One Text + natural wrap keeps the continuation flush. */} | ||
| {(kind === 'achieved' || kind === 'aborted') && lastReason?.trim() ? ( | ||
| {(kind === 'achieved' || kind === 'aborted' || kind === 'failed') && | ||
| lastReason?.trim() ? ( |
There was a problem hiding this comment.
This disjunction is growing — it now matches all terminal kinds except set, cleared, and checking. The same pattern appears in restoreGoal.ts:findLastTerminalGoal and goalCommand.ts:formatTerminalSummary. Consider extracting a shared isTerminalGoalKind() helper or a TERMINAL_GOAL_KINDS set so adding another terminal kind in the future only requires one change instead of N.
| return { decision: 'block', reason: continuationReasonForGoal(condition) }; | ||
| return { | ||
| decision: 'block', | ||
| reason: continuationReasonForGoal(condition), |
There was a problem hiding this comment.
The old code had a multi-line comment here explaining why the stop hook reason uses fixed text (continuationReasonForGoal) instead of passing through the judge's free-form diagnostic. That security rationale (untrusted transcript-derived text must not flow into the model's continuation prompt) is non-obvious and was removed in this reformatting. Worth preserving — new contributors may wonder why verdict.reason isn't included in the hook response.
| durationMs: Date.now() - latest.setAt, | ||
| lastReason: verdict.reason, | ||
| }); | ||
| return { continue: true }; |
There was a problem hiding this comment.
[Suggestion] failed terminal state missing systemMessage
The failed path returns { continue: true } with no systemMessage, while the aborted path (line 222) returns { continue: true, systemMessage: GOAL_ABORTED_REASON }. After a failed outcome, the user gets no explicit hint to rephrase and re-attempt the goal — unlike for aborted goals which include actionable guidance ("Re-set with /goal <condition> if you still need it").
Consider adding a GOAL_FAILED_REASON constant and including it as systemMessage in both the finishGoal call and the return, matching the aborted pattern:
const GOAL_FAILED_REASON =
'Goal determined to be impossible; cleared. Re-phrase and re-set with `/goal <condition>` to try a different approach.';— glm-5.1 via Qwen Code /review
|
|
||
| // Compile-time only: fails if JudgeResult grows a key that the response schema | ||
| // key list does not include. | ||
| const JUDGE_RESULT_SCHEMA_COVERS_INTERFACE: SchemaCoversJudgeResult = true; |
There was a problem hiding this comment.
[Suggestion] Compile-time assertion is exported but never imported
JUDGE_RESULT_SCHEMA_COVERS_INTERFACE is exported but never imported anywhere in the codebase. Its sole purpose is to trigger a compile-time type error when JUDGE_RESULT_SCHEMA_KEYS diverges from JudgeResult. An aggressive "remove unused exports" lint rule or manual cleanup could delete it, silently breaking the compile-time safety net.
Consider either:
- Keeping it unexported with a
voidstatement (which the diff already has) — drop theexportkeyword, or - Adding a comment explaining why it must be kept.
— glm-5.1 via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
No new issues found. Existing 7 open review comments already cover all concerns — no additional findings in this pass. LGTM! ✅ — DeepSeek/deepseek-v4-pro via Qwen Code /review
| ok, | ||
| reason: reasonText, | ||
| ...(impossible && !ok ? { impossible: true } : {}), | ||
| }; |
There was a problem hiding this comment.
[Suggestion] parseJudgeReply uses strict === true for impossible
const impossible = (payload as { impossible?: unknown }).impossible === true;If the model returns "true" (string) or 1 (number), the verdict is silently treated as not-impossible and the goal continues looping. The structured-output schema requires boolean, but parseJudgeReply already has fallback parsing for malformed JSON (bracket-search in start/end), suggesting the model doesn't always respect the schema.
Consider adding a tolerance test for non-boolean impossible values and optionally using Boolean() coercion to be defensive.
| }; | |
| const impossible = Boolean((payload as { impossible?: unknown }).impossible); |
— DeepSeek/deepseek-v4-pro via Qwen Code /review
| @@ -81,13 +85,20 @@ export const GoalStatusMessage: React.FC<GoalStatusMessageProps> = ({ | |||
| prefixColor: theme.text.secondary, | |||
| title: 'Goal cleared', | |||
There was a problem hiding this comment.
[Suggestion] GoalStatusMessage is a pure functional component but not wrapped in React.memo
In the Ink/React rendering model, every streaming event triggers a parent re-render, causing all historical GoalStatusMessage instances in the scrollback to re-render even though their props (kind, condition, iterations, durationMs, lastReason) are immutable once the card is rendered.
| title: 'Goal cleared', | |
| export const GoalStatusMessage: React.FC<GoalStatusMessageProps> = React.memo(({ |
— DeepSeek/deepseek-v4-pro via Qwen Code /review
| | 'checking'; | ||
|
|
||
| export const TERMINAL_GOAL_STATUS_KINDS = [ | ||
| 'achieved', |
There was a problem hiding this comment.
Nice. The satisfies readonly GoalStatusKind[] constraint plus isTerminalGoalStatusKind type guard means adding a new terminal kind to GoalTerminalKind without adding it here produces a type error at the assertNever call sites. Single source of truth for terminal-kind membership — exactly what I was suggesting.
| @@ -209,9 +239,12 @@ export function createGoalStopHookCallback(args: { | |||
| recordGoalIteration(sessionId, verdict.reason); | |||
| // Keep the judge's free-form diagnostic in goal state/UI only. The Stop | |||
| // hook reason is fed back to the model as the next continuation prompt, so | |||
There was a problem hiding this comment.
Thanks for restoring this. The security rationale (untrusted judge text must not flow into the continuation prompt) is non-obvious and worth preserving for future contributors.
| // Compile-time only: fails if JudgeResult grows a key that the response schema | ||
| // key list does not include. | ||
| const JUDGE_RESULT_SCHEMA_COVERS_INTERFACE: SchemaCoversJudgeResult = true; | ||
| void JUDGE_RESULT_SCHEMA_COVERS_INTERFACE; |
There was a problem hiding this comment.
Clean fix — unexported with void to suppress the unused-variable warning. The // Compile-time only comment makes the intent immediately clear.
Local validation reportVerified on 9f2430a in a clean checkout. Unit / typecheck
Spot check
GitHub CIAll 8 required checks green on 9f2430a: Lint / CodeQL ×2 / Test (mac+ubuntu+win) / Classify / Coverage skip. |
) Ported from QwenLM#4230, adapted to our goal subsystem (core/src/goal, single goalEvaluator) rather than upstream's goalJudge. The goal evaluator can now return a terminal "impossible" verdict — used only when the condition is genuinely unachievable in this session (self- contradictory, depends on an unavailable resource, or all reasonable approaches are exhausted). When it fires, the autonomous /goal loop stops and the goal is recorded as failed instead of continuing forever. Guardrails mirror upstream: the assistant claiming impossibility is evidence, not proof; "impossible" is only honoured alongside met:false; and an evaluator error or empty reply never fails a goal (stays met:false, continues). - types: `GoalEvaluationResult.impossible`, `GoalState.failedAt` - goalEvaluator: prompt + parser support the third verdict shape - GoalManager: `markImpossible(reason)` / `getLastFailedGoal()`; reset clears it - client: impossible verdict → markImpossible + stop (no continuation) - goalCommand: `/goal` status reports an abandoned-as-impossible goal, preferring the most recent terminal outcome - tests: parser, manager, and status-text coverage (60 goal tests pass) Co-authored-by: Automaker <automaker@localhost> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…sion, SDK canUseTool timeout) (#331) Documents behavior shipped in 0.51.x–0.52.0 that lacked coverage: - guides/goal.md: the evaluator's three verdicts (met / not met / impossible), the conservative impossible→abandoned terminal state, and how `/goal` status reports an abandoned goal. (QwenLM#4230) - explanation/agent-harness.md: new "Reactive compression" section — proactive vs. reactive compression and the one-shot force-compress-and-retry on a provider context-overflow rejection. (QwenLM#3879) - reference/sdk-api.md + contributing/sdk-typescript.md: document the `timeout.canUseTool` option and that the SDK now forwards it to the CLI so the control-plane timeout matches the callback's. (QwenLM#4491) (The `--max-tool-calls` / `--max-wall-time` budgets were already documented in guides/run-headless.md when those features shipped.) Co-authored-by: Automaker <automaker@localhost> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Summary
Validation
Commands run:
cd packages/core && npx vitest run src/goals/goalJudge.test.ts src/goals/goalHook.test.ts
cd packages/cli && npx vitest run src/ui/components/messages/GoalStatusMessage.test.tsx src/ui/commands/goalCommand.test.ts src/ui/utils/restoreGoal.test.ts
npm run build
npm run typecheck
git diff --check
Observed result: targeted tests passed, build passed, typecheck passed. npm run build reported an existing lint warning in packages/vscode-ide-companion/src/utils/editorGroupUtils.ts.
Scope / Risk
Linked Issues / Bugs
Related #4228