feat(sessions): cache-stable message assembly with volatile User-role tail#618
Merged
Conversation
… tail
Netclaw's LLM call assembly previously placed memory recall and dynamic
context layers as System-role messages immediately after the persisted
system prompt. These contain per-turn volatile content (memory recall,
current time, working-context), which meant the llama.cpp prompt cache
prefix match stopped at the boundary between the persisted prompt and
the dynamic layer — ~4867 tokens — on every turn. Conversation history
was never cached even when session-sticky routing pinned requests to
the same backend GPU.
Extract a pure-function SessionMessageAssembler that partitions the
outgoing message list into:
[0] System persisted prompt (unchanged)
[1] System static dynamic context OnceAtStart layers,
[session], [attachments]
[2..N] User/Assistant conversation history cacheable byte-stable prefix
[last] User volatile tail [memory-recall], current
time, [working-context],
slash command content,
overlay, turn restart notice
Result: on turn N+1, the longest common prefix with turn N's assembly
extends through all static content and all conversation history —
cache misses are confined to the new user message and its runtime
context tail.
Add SessionMessageAssemblerTests with 7 deterministic cache-stability
assertions including the core Prefix_extends_through_history check and
structural guards that prevent future changes from accidentally placing
volatile markers (current_utc, [memory-recall]) in the System prefix.
Add one actor-driven integration test in CompactionIntegrationTests
that drives two real turns and diffs FakeChatClient.ReceivedMessages
to catch regressions in the actor's wiring to the assembler.
Delete SessionRecallManager.InjectIntoMessages and LlmSessionActor's
InjectDynamicContextLayers private method — both are now dead code.
Closes #608
This was referenced Apr 12, 2026
Open
Aaronontheweb
added a commit
that referenced
this pull request
Apr 13, 2026
…NULL Two independent regressions surfaced together in production Slack sessions D0AC6CKBK5K/1776051715.090089 and D0AC6CKBK5K/1776075016.334849. 1. Tool-loop acknowledgement loop (PR #618 regression). SessionMessageAssembler emitted the volatile turn-context tail (memory recall, current time, working-context) as a ChatRole.User message at the end of the assembled list. During a tool loop, Qwen3's ChatML template read the trailing user-role block as a fresh user turn, so the model restarted its assistant response on every iteration, scanned back for the last real user content, and re-emitted "You're right - I had that backwards" before each tool call until context hit 262144/262144 and was force-compacted. Flip the role back to System; keep the placement at the end of the list so #618's cache stability win is preserved (llama.cpp KV cache is byte-level prefix matching, role tag at the end does not affect prefix stability). Update 4 existing SessionMessageAssembler tests for the role flip and add a new Volatile_tail_does_not_create_fake_user_turn_after_tool_result regression test that builds a mid-tool-loop history and asserts the trailing message is System-role with no User-role after the last Tool/Assistant message. 2. Legacy memory_anchors.domain NOT NULL constraint. PR #588 removed the Domain concept from the in-code schema but intentionally shipped no migration. CREATE TABLE IF NOT EXISTS is a no-op on existing databases, so production DBs still had `domain TEXT NOT NULL` on memory_anchors, memory_documents, memory_records, and memory_edges. Every new-anchor INSERT has been failing with SQLite Error 19 since the refactor, blocking memory curation entirely - no new memories have been written since 2026-04-12 20:37 despite successful distillation proposals. Add scripts/repair-memory-schema.sql, a one-off rename/rebuild/copy repair that drops the legacy column from all four tables while preserving rows. FTS5 virtual tables are standalone (no content= mode) and do not need touching. Also handle the AcceptedDistillationProposalsRecorded dead letter: the observer correctly replies to LlmSessionActor, but the existing handler only lived inside Passivating(). Add no-op Command<> registrations in Ready(), Processing(), and Compacting() so the informational reply does not land in DeadLetters during normal session states. Capture Sender into a local in SessionMemoryObserverActor.HandleRecordAcceptedDistillationProposals before calling Persist - the standard Akka.NET defense against Sender being overwritten by an interleaved message before the persist callback fires.
7 tasks
Aaronontheweb
added a commit
that referenced
this pull request
Apr 13, 2026
…NULL (#634) * fix(sessions): recover from volatile-tail loop and legacy domain NOT NULL Two independent regressions surfaced together in production Slack sessions D0AC6CKBK5K/1776051715.090089 and D0AC6CKBK5K/1776075016.334849. 1. Tool-loop acknowledgement loop (PR #618 regression). SessionMessageAssembler emitted the volatile turn-context tail (memory recall, current time, working-context) as a ChatRole.User message at the end of the assembled list. During a tool loop, Qwen3's ChatML template read the trailing user-role block as a fresh user turn, so the model restarted its assistant response on every iteration, scanned back for the last real user content, and re-emitted "You're right - I had that backwards" before each tool call until context hit 262144/262144 and was force-compacted. Flip the role back to System; keep the placement at the end of the list so #618's cache stability win is preserved (llama.cpp KV cache is byte-level prefix matching, role tag at the end does not affect prefix stability). Update 4 existing SessionMessageAssembler tests for the role flip and add a new Volatile_tail_does_not_create_fake_user_turn_after_tool_result regression test that builds a mid-tool-loop history and asserts the trailing message is System-role with no User-role after the last Tool/Assistant message. 2. Legacy memory_anchors.domain NOT NULL constraint. PR #588 removed the Domain concept from the in-code schema but intentionally shipped no migration. CREATE TABLE IF NOT EXISTS is a no-op on existing databases, so production DBs still had `domain TEXT NOT NULL` on memory_anchors, memory_documents, memory_records, and memory_edges. Every new-anchor INSERT has been failing with SQLite Error 19 since the refactor, blocking memory curation entirely - no new memories have been written since 2026-04-12 20:37 despite successful distillation proposals. Add scripts/repair-memory-schema.sql, a one-off rename/rebuild/copy repair that drops the legacy column from all four tables while preserving rows. FTS5 virtual tables are standalone (no content= mode) and do not need touching. Also handle the AcceptedDistillationProposalsRecorded dead letter: the observer correctly replies to LlmSessionActor, but the existing handler only lived inside Passivating(). Add no-op Command<> registrations in Ready(), Processing(), and Compacting() so the informational reply does not land in DeadLetters during normal session states. Capture Sender into a local in SessionMemoryObserverActor.HandleRecordAcceptedDistillationProposals before calling Persist - the standard Akka.NET defense against Sender being overwritten by an interleaved message before the persist callback fires. * refactor(sessions): collapse distillation ack no-op and trim test noise Review cleanup on top of the prior volatile-tail and NOT-NULL fixes. - LlmSessionActor: extract the three identical no-op `Command<AcceptedDistillationProposalsRecorded>(_ => { })` registrations from Ready/Processing/Compacting into a single `CommandDistillationAckNoOp()` helper with one canonical comment explaining why the non-passivation states need to swallow the observer's informational reply. Net: 3 copies of the handler and 3 near-duplicate comment blocks collapse to 1 helper + 1 comment + 3 single-line call sites. - SessionMessageAssemblerTests: * Recall_is_not_in_leading_system_prefix_even_when_resolved: drop the StringBuilder in favour of the per-message / break-on-non-System pattern already established by AssertNoVolatileContentInSystemPrefix. Same invariant, less allocation, same shape as the rest of the file. * Prefix_extends_through_history_when_startup_layers_settled, Volatile_tail_message_is_System_role_at_end_of_list, and Working_context_update_does_not_poison_system_prefix: trim WHAT- narrating comments that the identifiers and assertions already convey. Load-bearing WHY comments on the new regression test, the SessionMessageAssembler XML doc, and the SessionMemoryObserverActor Sender-capture rationale are preserved. No behavioural change. Verified with `dotnet test --filter "FullyQualifiedName~Sessions"` (253/253 passing) and `dotnet slopwatch analyze` (0 issues).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #608.
Netclaw's LLM message assembly previously placed memory recall and dynamic context layers as System-role messages immediately after the persisted system prompt. These contain volatile per-turn content (memory recall, current time, working-context), which meant the llama.cpp prompt-cache prefix match stopped at the boundary between the persisted prompt and the dynamic layer — ~4867 tokens — on every turn. Conversation history was never cached, even with session-sticky routing (#610) pinning requests to the same backend GPU.
The post-#610 eval baseline showed exactly this:
multi_turn_python_appT4 saw only a 6% prompt_ms improvement (1959ms → 1834ms) vs the ~500ms target.The fix
Extract a pure-function
SessionMessageAssemblerthat partitions the outgoing message list so cache-stable content comes first and volatile content is consolidated into a single User-role tail message:On turn N+1, the longest common prefix with turn N's assembly now extends through all static content and all conversation history that existed at turn N. Cache misses are confined to the new user turn and its runtime context tail. Each additional turn grows the cacheable prefix by exactly one user/assistant pair.
Deterministic regression guard
Aaron explicitly requested a cache-poisoning assertion that runs as a fast unit test — no LLM round-trip.
SessionMessageAssemblerTestsadds seven pure-function tests:Prefix_is_stable_across_turns_for_same_session— longest common prefix check with a turn-specific recall marker in each turn (sanity assertion that recall actually appears in the volatile tail, and that turn 1's marker does not leak into turn 2's assembly)Prefix_extends_through_history_when_startup_layers_settled— steady-state cache prefix ≥ 3 messagesVolatile_tail_message_is_User_role_at_end_of_listStatic_block_contains_session_id_and_attachment_hintWorking_context_update_does_not_poison_system_prefixRecall_is_not_in_system_prefix_even_when_resolvedVolatile_tail_is_suppressed_when_emptyA future change that accidentally places volatile content in an early message will fail these tests loudly.
Also added an actor-driven integration regression (
Cache_prefix_is_stable_across_two_turns_in_same_session) that drives two real turns throughLlmSessionActorand diffsFakeChatClient.ReceivedMessages[0]vs[1]— this catches regressions where the actor's wiring to the assembler gets broken even if the helper itself is correct.Cleanups
SessionRecallManager.InjectIntoMessages(33 lines, now dead code)LlmSessionActor.InjectDynamicContextLayersprivate method (82 lines)AttachmentContextHintconstant fromLlmSessionActor→SessionMessageAssemblerso it lives with its only consumerExpected impact
Rerunning the Docker eval suite against the Caddy-sticky llama.cpp endpoint should show:
multi_turn_python_appT4cached_tokensgrowing beyond the 4867 floor (T2 caches T1, T3 caches T1+T2, T4 caches T1+T2+T3)prompt_msdropping from ~1834ms toward the sub-1000ms range--cache-reuse Nto the llama-server systemd unit (separate testlab-setup change)Managed providers (Anthropic, OpenAI, OpenRouter) that do their own prefix caching benefit for the same reason: a stable system prefix plus cacheable conversation history means every turn after the first is mostly a cache hit.
Verification
Test plan
./evals/run-evals.shagainst the existing Caddy-sticky llama-server endpoint; confirmmulti_turn_python_appT4 showscached_tokens > 4867andprompt_ms < 1500ms