Skip to content

feat(sessions): structured compaction summary with monotonic boundary#597

Merged
Aaronontheweb merged 2 commits into
devfrom
compaction-rework
Apr 11, 2026
Merged

feat(sessions): structured compaction summary with monotonic boundary#597
Aaronontheweb merged 2 commits into
devfrom
compaction-rework

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Summary

Rework session compaction based on source-level reads of four production LLM harnesses:

Harness Commit
OpenCode (SST, TypeScript) sst/opencode@2719063
Aider (Python) Aider-AI/aider@f09d706
Cline (TypeScript / VS Code) cline/cline@a0faf7c
Claude Code (docs only) code.claude.com/docs/en/*

Three adopted ideas:

  1. Cline's 9-section structured summary template, including the explicit anti-drift rule in the Task Evolution section (direct quotes from user messages to prevent drift after context compacting). Replaces the free-form bullet list that was losing grounding across successive compactions.
  2. OpenCode's truncate-only-at-user-message-boundaries rule, enforced in ExtractiveSessionReducer via a backward walk. Guarantees tool call/result pair integrity, which the openspec netclaw-session "pair integrity" requirement already mandated but the code did not actually enforce.
  3. Self-session-id disambiguation in the observer system prompt so the observer can explicitly mark foreign session IDs from tool-call history. This is the Slack failure that triggered the whole investigation (2026-04-11) — ArdyBot lost its own session ID after a compaction that followed a turn investigating another session.

Also adds SessionState.CompactionBoundaryIndex as monotonic metadata pointing at the most recent [session-summary session:{id}] marker. Informational for MVP; the infrastructure supports future checkpoint rollback and multi-tier compaction without a data migration.

Summary messages now use a [session-summary session:{id}] header. The observer prompt tells the model to preserve any prior summary block verbatim on successive compactions — belt-and-suspenders defense against summary-over-summary decay paired with the reducer's backward walk that won't chop a prior summary.

Journal compatibility: SessionCompacted events written before this change deserialize cleanly. The new CompactionBoundaryIndex field is optional and Apply(SessionCompacted) defaults to the post-system offset when absent.

OpenSpec

This PR ships the full OpenSpec change at openspec/changes/compaction-rework/ (proposal, design, netclaw-session delta spec, tasks).

Related

  • Stacked under: Aaronontheweb/netclaw#TBD (durable WorkingContext grounding — PR2)
  • Follow-up: Aaronontheweb/netclaw#595 (session CWD tracking + project identity re-read, milestone 0.12)
  • Follow-up: Aaronontheweb/netclaw#596 (authoritative session CWD for path-taking tools, milestone 0.12)

Test plan

  • dotnet build src/Netclaw.Actors.Tests/Netclaw.Actors.Tests.csproj — zero warnings
  • dotnet test src/Netclaw.Actors.Tests/Netclaw.Actors.Tests.csproj890/890 pass
  • dotnet slopwatch analyze — clean
  • New unit tests in ObservationPromptBuilderTests.cs: 9-section layout, self-session-id, preserve-prior-summary rule, direct-quotes-in-task-evolution
  • New unit tests in ExtractiveSessionReducerTests.cs: backward walk to user boundary, skip system nudges, keep-zero edge case, degenerate fallback
  • Existing CompactionIntegrationTests.cs cases pass (session-id header, observer self-session-id, second-compaction buffer drain, session recovery after compaction+kill)
  • Manual reproduction of the original Slack failure (requires a local daemon and a multi-session Slack thread; deferred to integration testing against a running instance)

@Aaronontheweb Aaronontheweb added this to the 0.12 milestone Apr 11, 2026
@Aaronontheweb Aaronontheweb added context-pipeline LLM context assembly: prompt layers, dynamic injection, memory recall, temporal grounding enhancement New feature or request sessions LLM session actor, turn lifecycle, pipelines labels Apr 11, 2026
Rework session compaction based on source-level reads of four production
LLM harnesses (Aider f09d706, sst/opencode 2719063, cline/cline a0faf7c,
Claude Code docs). Three adopted ideas:

- Cline's 9-section structured summary template with anti-drift rule
  (Task Evolution section requires direct quotes from user messages) to
  replace the free-form bullet list that was losing grounding across
  successive compactions
- OpenCode's truncate-only-at-user-message-boundaries rule, enforced in
  ExtractiveSessionReducer via backward walk — guarantees tool call/
  result pair integrity which the openspec netclaw-session "pair
  integrity" requirement already mandated but the code did not enforce
- Self-session-id disambiguation in the observer system prompt so the
  observer can mark foreign session IDs from tool-call history (the
  failure mode that triggered this rework — ArdyBot lost its own
  session ID after a compaction that followed a turn investigating
  another session)

Summary messages now use a `[session-summary session:{id}]` header.
This is the only recognition marker: consumers (observer, reducer, UI)
walk history looking for the header prefix rather than consulting a
separately-persisted index. The observer prompt tells the model to
preserve any prior summary block verbatim on successive compactions —
belt-and-suspenders defense against summary-over-summary decay paired
with the reducer's backward walk that won't chop a prior summary.

Journal compatibility: no new fields on `SessionCompacted` or
`SessionSnapshot`. Old-format compactions (User-role `[observations
from earlier in this session]` messages) still deserialize cleanly and
are folded into the new format by the next compaction via
`WrapObservations`.

Refs Aaronontheweb/netclaw#595 (session CWD tracking + project
identity re-read) and #596 (authoritative session CWD for path-taking
tools) which are tracked follow-ups on milestone 0.12.

Research and architectural discussion captured in OpenSpec change
artifacts at openspec/changes/compaction-rework/. Working-context
grounding (durable RecentFiles / OpenGoals / ProgressMarkers on
SessionState) ships as a separate OpenSpec change in PR2.

Tests: 890/890 pass. Slopwatch clean.
@Aaronontheweb Aaronontheweb merged commit f4868c2 into dev Apr 11, 2026
3 of 4 checks passed
@Aaronontheweb Aaronontheweb deleted the compaction-rework branch April 11, 2026 17:36
Aaronontheweb added a commit that referenced this pull request Apr 11, 2026
Follow-up to #597: sync the compaction-rework delta spec into the
canonical netclaw-session spec and move the change directory to the
archive. Consolidates the stale 2026-03-25 refactor-llm-session-actor
delta fragment for "Conversation compaction" into the canonical
requirement — that fragment had been sitting un-merged since the earlier
refactor change was archived, and this rewrite supersedes its content.
"Decoupled immutable session state" fragment from the same archive is
left alone (unrelated to compaction work).
Aaronontheweb added a commit that referenced this pull request Apr 11, 2026
Stacks on the compaction-rework change (merged as #597). Adds a
`WorkingContext` field to `SessionState` that tracks recent files the
agent has read/written/edited, survives compaction, actor recovery,
and daemon restart without depending on the observer LLM to
reconstruct it.

This introduces a new tier of session state that sits parallel to
existing tiers:

- Skills: stateless how-to reference (already exists)
- Memory: cross-session facts (already exists, via memory store)
- WorkingContext: in-session mutable state (new)
- Conversation: turn history (already exists)

`WorkingContext` carries a single field: `RecentFiles` — a bounded
ring buffer of 10 file paths, most-recent-first, deduped on repeat
access. Updated automatically by `LlmSessionActor` when a path-taking
tool (file_read, file_write, file_edit, etc.) completes. The path is
extracted from the tool call's `ArgumentsJson` via
`WorkingContextUpdater.TryExtractFilePath` which probes well-known
JSON field names (`path`, `file_path`, `filePath`, `file`, `filename`,
`fileName`) — no tool-name allowlist, so first-party tools and MCP
filesystem tools participate automatically.

Security: `AddRecentFile` rejects paths containing control characters
(newline, carriage return, null byte). Without this, an attacker-
crafted path containing a literal newline would break out of the
`recent_files:` section in the `[working-context]` block and inject
arbitrary content into the LLM's system prompt. Rejected paths are
silently dropped at the earliest point in the ingestion pipeline.

Injection: `WorkingContext` is emitted as a `[working-context]` block
via `InjectDynamicContextLayers` on every LLM call when non-empty,
adjacent to the existing `[session]` block. Suppressed entirely when
`WorkingContext.IsEmpty` so the model doesn't see a barren header.

Explicitly NOT in this change:
- OpenGoals / ProgressMarkers fields and the observer-output parser
  that would populate them — deferred per "no hypothetical future
  requirements" rule. Can be added in a follow-up change when the
  parser lands.
- CurrentWorkingDirectory / ActiveProjectPath — GitHub #595.
- Authoritative CWD for path-taking tools — GitHub #596.

Journal compatibility: `WorkingContext` is a new optional field on
`SessionSnapshot` (ProtoMember 6) and `SessionCompacted` (ProtoMember
7). ProtoMember 5/6 respectively are reserved holes — formerly
`CompactionBoundaryIndex`, removed before compaction-rework merged.
Old snapshots and events without the field deserialize to
`WorkingContext.Empty` via `SessionState.FromSnapshot` /
`Apply(SessionCompacted)`.

Precedent: Cline's `TaskState.currentFocusChainChecklist` in
`src/core/task/focus-chain/` is the closest comparable pattern — a
persistent markdown todo list tracked on disk, re-attached to every
LLM call, explicitly protected from re-summarization. `WorkingContext`
is the Netclaw equivalent, stored on `SessionState` rather than a
standalone file.

Tests: 933/933 pass (+11 new). Slopwatch clean (SW003 on the
JsonException catch is explicitly ignored via inline directive with
justification — LLM-generated tool args can be malformed and "no
path tracked" is the correct handling).

Refs Aaronontheweb/netclaw#595, Aaronontheweb/netclaw#596
Aaronontheweb added a commit that referenced this pull request Apr 11, 2026
Stacks on the compaction-rework change (merged as #597). Adds a
`WorkingContext` field to `SessionState` that tracks recent files the
agent has read/written/edited, survives compaction, actor recovery,
and daemon restart without depending on the observer LLM to
reconstruct it.

This introduces a new tier of session state that sits parallel to
existing tiers:

- Skills: stateless how-to reference (already exists)
- Memory: cross-session facts (already exists, via memory store)
- WorkingContext: in-session mutable state (new)
- Conversation: turn history (already exists)

`WorkingContext` carries a single field: `RecentFiles` — a bounded
ring buffer of 10 file paths, most-recent-first, deduped on repeat
access. Updated automatically by `LlmSessionActor` when a path-taking
tool (file_read, file_write, file_edit, etc.) completes. The path is
extracted from the tool call's `ArgumentsJson` via
`WorkingContextUpdater.TryExtractFilePath` which probes well-known
JSON field names (`path`, `file_path`, `filePath`, `file`, `filename`,
`fileName`) — no tool-name allowlist, so first-party tools and MCP
filesystem tools participate automatically.

Security: `AddRecentFile` rejects paths containing control characters
(newline, carriage return, null byte). Without this, an attacker-
crafted path containing a literal newline would break out of the
`recent_files:` section in the `[working-context]` block and inject
arbitrary content into the LLM's system prompt. Rejected paths are
silently dropped at the earliest point in the ingestion pipeline.

Injection: `WorkingContext` is emitted as a `[working-context]` block
via `InjectDynamicContextLayers` on every LLM call when non-empty,
adjacent to the existing `[session]` block. Suppressed entirely when
`WorkingContext.IsEmpty` so the model doesn't see a barren header.

Explicitly NOT in this change:
- OpenGoals / ProgressMarkers fields and the observer-output parser
  that would populate them — deferred per "no hypothetical future
  requirements" rule. Can be added in a follow-up change when the
  parser lands.
- CurrentWorkingDirectory / ActiveProjectPath — GitHub #595.
- Authoritative CWD for path-taking tools — GitHub #596.

Journal compatibility: `WorkingContext` is a new optional field on
`SessionSnapshot` (ProtoMember 6) and `SessionCompacted` (ProtoMember
7). ProtoMember 5/6 respectively are reserved holes — formerly
`CompactionBoundaryIndex`, removed before compaction-rework merged.
Old snapshots and events without the field deserialize to
`WorkingContext.Empty` via `SessionState.FromSnapshot` /
`Apply(SessionCompacted)`.

Precedent: Cline's `TaskState.currentFocusChainChecklist` in
`src/core/task/focus-chain/` is the closest comparable pattern — a
persistent markdown todo list tracked on disk, re-attached to every
LLM call, explicitly protected from re-summarization. `WorkingContext`
is the Netclaw equivalent, stored on `SessionState` rather than a
standalone file.

Tests: 933/933 pass (+11 new). Slopwatch clean (SW003 on the
JsonException catch is explicitly ignored via inline directive with
justification — LLM-generated tool args can be malformed and "no
path tracked" is the correct handling).

Refs Aaronontheweb/netclaw#595, Aaronontheweb/netclaw#596
Aaronontheweb added a commit that referenced this pull request Apr 11, 2026
Stacks on the compaction-rework change (merged as #597). Adds a
`WorkingContext` field to `SessionState` that tracks recent files the
agent has read/written/edited, survives compaction, actor recovery,
and daemon restart without depending on the observer LLM to
reconstruct it.

This introduces a new tier of session state that sits parallel to
existing tiers:

- Skills: stateless how-to reference (already exists)
- Memory: cross-session facts (already exists, via memory store)
- WorkingContext: in-session mutable state (new)
- Conversation: turn history (already exists)

`WorkingContext` carries a single field: `RecentFiles` — a bounded
ring buffer of 10 file paths, most-recent-first, deduped on repeat
access. Updated automatically by `LlmSessionActor` when a path-taking
tool (file_read, file_write, file_edit, etc.) completes. The path is
extracted from the tool call's `ArgumentsJson` via
`WorkingContextUpdater.TryExtractFilePath` which probes well-known
JSON field names (`path`, `file_path`, `filePath`, `file`, `filename`,
`fileName`) — no tool-name allowlist, so first-party tools and MCP
filesystem tools participate automatically.

Security: `AddRecentFile` rejects paths containing control characters
(newline, carriage return, null byte). Without this, an attacker-
crafted path containing a literal newline would break out of the
`recent_files:` section in the `[working-context]` block and inject
arbitrary content into the LLM's system prompt. Rejected paths are
silently dropped at the earliest point in the ingestion pipeline.

Injection: `WorkingContext` is emitted as a `[working-context]` block
via `InjectDynamicContextLayers` on every LLM call when non-empty,
adjacent to the existing `[session]` block. Suppressed entirely when
`WorkingContext.IsEmpty` so the model doesn't see a barren header.

Explicitly NOT in this change:
- OpenGoals / ProgressMarkers fields and the observer-output parser
  that would populate them — deferred per "no hypothetical future
  requirements" rule. Can be added in a follow-up change when the
  parser lands.
- CurrentWorkingDirectory / ActiveProjectPath — GitHub #595.
- Authoritative CWD for path-taking tools — GitHub #596.

Journal compatibility: `WorkingContext` is a new optional field on
`SessionSnapshot` (ProtoMember 6) and `SessionCompacted` (ProtoMember
7). ProtoMember 5/6 respectively are reserved holes — formerly
`CompactionBoundaryIndex`, removed before compaction-rework merged.
Old snapshots and events without the field deserialize to
`WorkingContext.Empty` via `SessionState.FromSnapshot` /
`Apply(SessionCompacted)`.

Precedent: Cline's `TaskState.currentFocusChainChecklist` in
`src/core/task/focus-chain/` is the closest comparable pattern — a
persistent markdown todo list tracked on disk, re-attached to every
LLM call, explicitly protected from re-summarization. `WorkingContext`
is the Netclaw equivalent, stored on `SessionState` rather than a
standalone file.

Tests: 933/933 pass (+11 new). Slopwatch clean (SW003 on the
JsonException catch is explicitly ignored via inline directive with
justification — LLM-generated tool args can be malformed and "no
path tracked" is the correct handling).

Refs Aaronontheweb/netclaw#595, Aaronontheweb/netclaw#596
Aaronontheweb added a commit that referenced this pull request Apr 11, 2026
* feat(sessions): durable WorkingContext grounding

Stacks on the compaction-rework change (merged as #597). Adds a
`WorkingContext` field to `SessionState` that tracks recent files the
agent has read/written/edited, survives compaction, actor recovery,
and daemon restart without depending on the observer LLM to
reconstruct it.

This introduces a new tier of session state that sits parallel to
existing tiers:

- Skills: stateless how-to reference (already exists)
- Memory: cross-session facts (already exists, via memory store)
- WorkingContext: in-session mutable state (new)
- Conversation: turn history (already exists)

`WorkingContext` carries a single field: `RecentFiles` — a bounded
ring buffer of 10 file paths, most-recent-first, deduped on repeat
access. Updated automatically by `LlmSessionActor` when a path-taking
tool (file_read, file_write, file_edit, etc.) completes. The path is
extracted from the tool call's `ArgumentsJson` via
`WorkingContextUpdater.TryExtractFilePath` which probes well-known
JSON field names (`path`, `file_path`, `filePath`, `file`, `filename`,
`fileName`) — no tool-name allowlist, so first-party tools and MCP
filesystem tools participate automatically.

Security: `AddRecentFile` rejects paths containing control characters
(newline, carriage return, null byte). Without this, an attacker-
crafted path containing a literal newline would break out of the
`recent_files:` section in the `[working-context]` block and inject
arbitrary content into the LLM's system prompt. Rejected paths are
silently dropped at the earliest point in the ingestion pipeline.

Injection: `WorkingContext` is emitted as a `[working-context]` block
via `InjectDynamicContextLayers` on every LLM call when non-empty,
adjacent to the existing `[session]` block. Suppressed entirely when
`WorkingContext.IsEmpty` so the model doesn't see a barren header.

Explicitly NOT in this change:
- OpenGoals / ProgressMarkers fields and the observer-output parser
  that would populate them — deferred per "no hypothetical future
  requirements" rule. Can be added in a follow-up change when the
  parser lands.
- CurrentWorkingDirectory / ActiveProjectPath — GitHub #595.
- Authoritative CWD for path-taking tools — GitHub #596.

Journal compatibility: `WorkingContext` is a new optional field on
`SessionSnapshot` (ProtoMember 6) and `SessionCompacted` (ProtoMember
7). ProtoMember 5/6 respectively are reserved holes — formerly
`CompactionBoundaryIndex`, removed before compaction-rework merged.
Old snapshots and events without the field deserialize to
`WorkingContext.Empty` via `SessionState.FromSnapshot` /
`Apply(SessionCompacted)`.

Precedent: Cline's `TaskState.currentFocusChainChecklist` in
`src/core/task/focus-chain/` is the closest comparable pattern — a
persistent markdown todo list tracked on disk, re-attached to every
LLM call, explicitly protected from re-summarization. `WorkingContext`
is the Netclaw equivalent, stored on `SessionState` rather than a
standalone file.

Tests: 933/933 pass (+11 new). Slopwatch clean (SW003 on the
JsonException catch is explicitly ignored via inline directive with
justification — LLM-generated tool args can be malformed and "no
path tracked" is the correct handling).

Refs Aaronontheweb/netclaw#595, Aaronontheweb/netclaw#596

* feat(sessions): observability + compaction-pipeline integration test for WorkingContext

WorkingContextUpdater now takes an optional ILoggingAdapter and emits debug
logs for both tracked files and schema-drift (tool call had string-valued
arguments but none of the probed field names matched). Operators hunting
"why isn't this file in [working-context]" get a concrete signal in debug
logs.

Adds an end-to-end integration test that drives a file_read tool call
through the real compaction pipeline (3 turns: populate → compact → probe)
and verifies the [working-context] block still contains src/Rect.cs after
compaction. This exercises WorkingContextUpdater, the SessionCompacted
event path, and InjectDynamicContextLayers together — not just direct
Apply() on an in-memory state.

Refs #600 (sub-agent gap discovered during this work).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

context-pipeline LLM context assembly: prompt layers, dynamic injection, memory recall, temporal grounding enhancement New feature or request sessions LLM session actor, turn lifecycle, pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant