Skip to content

Compaction always fails to generate summaries - falls back to static text #3479

@fdsouvenir

Description

@fdsouvenir

Compaction always fails to generate summaries - "Summary unavailable due to context limits"

Bug Description

Compaction summaries are never generated. Every compaction entry in session
JSONL contains the fallback text "Summary unavailable due to context limits.
Older messages were truncated." with no actual summary content.

This causes complete context loss after compaction - the agent loses all
memory of prior conversation.

Evidence

In a Discord channel session with compactionCount: 2:

  • 7 compaction entries in the session JSONL, ALL using FALLBACK_SUMMARY
  • tokensBefore values: ~181K, 44, ~180K, ~195K, 166, ~184K, 269
  • No entry contains an actual AI-generated summary
  • Only preserved data: file operation lists from safeguard extension

Note the pattern: compaction entries come in pairs - one with high token counts
(~180-195K, the actual session content) and one with very low token counts
(44-269, likely the post-compaction state being re-compacted immediately).

Expected Behavior

Compaction should produce a meaningful summary of conversation history,
preserving key context, decisions, and file operations.

Actual Behavior

summarizeInStages() throws an error caught by the safeguard extension's
catch block at compaction-safeguard.js:202, falling back to:
"Summary unavailable due to context limits. Older messages were truncated."

Only file operation lists are preserved (from safeguard details).

Root Cause Analysis

  1. Reactive-only compaction: Auto-compaction only triggers AFTER context
    overflow, meaning sessions are at ~180-195K of 200K when compaction starts.
    At this point, the messages to summarize may be too large for even the
    chunked summarization approach.

  2. Chunk sizing may be insufficient: With BASE_CHUNK_RATIO = 0.4,
    chunks are 40% of context (80K tokens). The summarization API call needs
    prompt + chunk + output space. If chunk is 80K in a 200K window, there
    may not be enough room for the summarization prompt and output.

  3. Fallback cascade fails completely: summarizeWithFallback tries
    full → partial (without oversized) → text fallback. If even partial
    fails, the only output is a count of messages. But the exception
    propagates up to the safeguard's catch block before reaching the
    text fallback.

  4. pruneHistoryForContextShare drops without summarizing: At
    compaction-safeguard.js:145, when new content > 50% of context, oldest
    chunks are dropped entirely - no attempt to summarize them first.

  5. Double-compaction pattern: Compaction fires twice in quick succession.
    The first fails to summarize and falls back, then a second compaction
    fires on the tiny residual (44-269 tokens) and also fails.

Suggested Fixes

  1. Proactive compaction: Trigger compaction at ~60-70% context usage
    instead of waiting for overflow. This gives summarization room to work.

  2. Smaller chunks for summarization: The 40% base ratio is too
    aggressive. Consider 20-25% to leave room for prompt + output.

  3. Use cheaper/smaller model for summarization: Summarization doesn't
    need the session model. Using Sonnet or Haiku would be faster, cheaper,
    and less likely to hit rate limits.

  4. Better fallback: When summarization fails, at minimum extract
    user messages and produce a bullet-point list of topics discussed.

  5. Prevent double-compaction: Add a guard to prevent compaction from
    triggering again immediately after a failed compaction.

  6. Improved logging: Log chunk sizes attempted, adaptive ratio computed,
    pruned chunk counts, and the specific API error code/status.

Environment

  • Model: claude-opus-4-5
  • Context window: 200K tokens
  • Compaction mode: safeguard
  • Session type: Discord channel (long-running)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleMarked as stale due to inactivity

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions