Skip to content

[Bug]: Compaction summary 'Remaining Work' sections silently expand scope of vague but consistent user instructions #38364

@ferminquant

Description

@ferminquant

Bug Description

After context compaction, the model treats the summary's ## Remaining Work / ## Pending User Asks sections as implicit scope expansion when the latest user message is topically consistent but vague. The SUMMARY_PREFIX (fixed in #35383) correctly handles contradictions and overt topic changes, but when the user says something like "fix the issue following the new steps" — vague but on-topic — the compaction body's actionable-sounding sections fill in the blanks. The model then executes a conflated task that exceeds what the user asked and may violate active skill filters.

Specific failure chain observed:

  1. A session lineage compacts, producing a summary with ## Remaining Work listing genuinely unfinished items (e.g. "check 8 PRs, investigate #89183").
  2. The user returns with a new instruction that is topically consistent but vague (e.g. "fix the issue following these new steps").
  3. The SUMMARY_PREFIX says "if consistent, use as background" — which the model interprets as permission to use the ## Remaining Work items as the concrete task definition.
  4. The model conflates separate items into one query, violating skill-level filters (e.g. --draft=false) because the compaction's scope expansion overrides the skill's explicit filter.

The bug produces a report that appears correct to the user (all results are real) but violates the skill's explicit command — a silent scope expansion that's hard to catch because nothing visibly errors.

Steps to Reproduce

  1. Start a long session working on multiple PRs in a repo. Let the session produce genuinely unfinished items (e.g. "check PRs, investigate draft PR").
  2. Let automatic context compaction fire, producing a summary with ## Remaining Work and ## Pending User Asks sections listing those items.
  3. Return to the session and send a new message about the same general topic but with a vague instruction that could encompass the old items (e.g. "fix the issue following the new steps").
  4. Load a skill that has a specific filter (e.g. --draft=false) and execute it.
  5. Observe: the skill's explicit filter is violated because the compaction's "Remaining Work" expanded the scope before the skill was loaded.

Expected Behavior

SUMMARY_PREFIX's "latest message WINS" rule should cover not just overt contradictions but also scope creep. When the latest user message is vague, the model should not use ## Remaining Work / ## Pending User Asks compaction sections to fill in the specifics. If the user's instruction is ambiguous, the model should either ask for clarification or limit execution to only what the skill explicitly commands. The skill's filter (e.g. --draft=false) must always win over compaction scope.

Actual Behavior

Compaction ## Remaining Work sections silently expand the task scope. The model runs a broader query than the skill specifies, and the violation goes undetected because the output looks correct (it just includes more than it should).

Affected Component

  • Agent Core (conversation loop, context compression, memory)

Debug Report

N/A — internal model behavior, not a gateway deployment issue. The fix for #35344 updated the prefix but the body sections still carry actionable weight.

Operating System

N/A — cross-platform (observed on WSL Ubuntu)

Root Cause Analysis

The fix in #35383 correctly rewrote SUMMARY_PREFIX to say "latest message WINS — discard those stale items entirely." However, the prefix's carveout for consistency ("If the latest user message is consistent... you may use the summary as background") creates a trap:

  • When the user changes topic → WINS triggers correctly, old items discarded.
  • When the user gives a vague but topically consistent instruction → the ## Remaining Work sections look like "consistent background" and the model uses them to flesh out the goal.

The root cause is twofold:

  1. The compaction summary's body sections are generated by the LLM summarizer with actionable framing (## Remaining Work, ## Pending User Asks) that mirrors real task-management language.
  2. The model's natural behavior is to treat structured task-like sections as actionable — format beats framing. The SUMMARY_PREFIX's meta-instruction is weaker than the structured body that follows it.

Related Issues — Non-overlap explanation

Proposed Fix

Options (not mutually exclusive):

  1. Remove ## Remaining Work and ## Pending User Asks from the compaction summary template — these are the sections that trigger the misleading framing. Replace them with less actionable phrasing like ## Historical Context or ## Previous Work.

  2. Strengthen the compaction handoff body itself — add a structural marker after ## Remaining Work / ## Pending User Asks that the model cannot easily override, e.g. appending a separator like --- These items are historical and should not be treated as active tasks --- directly after each actionable section.

  3. Add a pre-flight assertion in the agent loop — before executing any skill, check whether the model's planned query scope matches the skill's explicit filters. If the scope is wider, flag it.

The cleanest fix is option 1: rename the summary body sections to purely historical framing so the model cannot read them as active instructions even when the user's message is vague.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High — major feature broken, no workaroundcomp/agentCore agent loop, run_agent.py, prompt buildertype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions