fix(#41607): Strengthen compression summary to prevent stale instructions execution#41634
Closed
iamlukethedev wants to merge 1 commit into
Closed
Conversation
…revent stale instructions execution Addresses issue where stale '## Pending User Asks' and '## Remaining Work' sections in compression summaries were incorrectly executed as current tasks when they had topical overlap with the latest user message. Changes: 1. STRENGTHENED PREAMBLE: Rewrote SUMMARY_PREFIX with explicit 'STALE — DISCARD' language, replacing weak 'treat as background reference' with: -⚠️ CRITICAL RULE section explicitly listing dangerous section names - Bullet points identifying stale items (Pending Asks, Remaining Work, In Progress, Blocked, Completed Actions) - Any phrasing that sounds like 'next steps' or 'to-do' 2. NEGATIVE EXAMPLES: Added three concrete before/after examples showing: - Wrong approach (acting on stale summary items) - Correct approach (discarding stale items, responding to latest message) - Covers auth, API endpoint, and multi-priority scenarios 3. CLARITY ON MEMORY vs SUMMARY: Explicitly contrasted: - Memory (MEMORY.md, USER.md) = eternal, authoritative - Summary sections = transient, must be discarded 4. RENAMED DANGEROUS SECTIONS in fallback template: - '## Pending User Asks' → '## Protected Context (Background Reference Only)' - '## Remaining Work' → '## Session Context (fallback only)' 5. UPDATED LLM SUMMARIZER TEMPLATE to prevent creation of dangerous sections: - Replaced '## Pending User Asks' section with '## Protected Context' - Removed '## Remaining Work' (fold into Critical Context) - Added explicit instruction: 'Do NOT phrase as pending or remaining — phrase as completed background facts' - Added critical warning: 'Do not create a ## Pending User Asks section or anything that looks like unfinished work — the model will execute it' Root cause: The previous preamble's soft language ('treat as background reference') was insufficient when summary items had topical overlap with the latest message. Models would conflate stale summary items with current context. This fix makes the discard rule explicit, unavoidable, and surrounded by concrete examples so the intent is unambiguous. Fixes NousResearch#41607
Collaborator
Author
|
@alt-glitch if this is a duplicate, should I close my PR? |
Contributor
|
Closing in favor of PR #44454 (merged), which combines #44345's heading constants with #41650's carveout removal plus a frozen-prefix fixup for backward compatibility. Your PR targeted the same stale-instruction class — thanks for the work and the candor about overlap; the negative-example ideas in your rewrite informed the review even though we went with the constant-based approach. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #41607
Problem: After context compression, stale instructions from the summary (especially '## Pending User Asks' and '## Remaining Work' sections) were being executed as current tasks when they had topical overlap with the latest user message, even though the preamble said to discard them.
Root Cause: The previous preamble's soft language ('treat as background reference') was insufficient. When summary items had topical overlap with the latest message, models would conflate stale summary items with current context.
Solution: Implemented all 4 suggestions from the issue:
Strengthened SUMMARY_PREFIX preamble with explicit 'STALE — DISCARD' language
Added negative examples showing before/after for three scenarios
Clarified memory vs summary distinction
Renamed dangerous sections to prevent execution-as-task misreading
Testing: Verified SUMMARY_PREFIX compiles and renders correctly. All changes are safe text updates to guidance and templates — no behavioral changes to compression logic itself.
Impact: Prevents the model from acting on stale summary items even when topical overlap makes them look relevant. Explicit negative examples make the intent unambiguous.