Summary
The SessionMemoryObserverActor prevents its parent LlmSessionActor from ever passivating. Sessions stay alive indefinitely — we observed session D0AC6CKBK5K/1774412603.599179 running for 13+ hours with 435 distillation cycles, the vast majority of which did nothing.
Root cause
The observer uses Context.SetReceiveTimeout(idleTimeout) (90s) to trigger periodic distillation. When there's no new content, TriggerDistillation sends SessionDistillationCompleted.Empty back to the parent:
// SessionMemoryObserverActor.cs:145-149
if (!_hasNewContent)
{
replyTo.Tell(SessionDistillationCompleted.Empty); // ← this resets parent's idle timer
return;
}
In Akka.NET, any received message (except ReceiveTimeout itself) resets the actor's receive timeout timer. The parent's 30-minute idle passivation timeout (Context.SetReceiveTimeout(_config.IdleTimeout)) gets reset every 90 seconds by these empty distillation results, so the session never passivates.
The feedback loop
Observer idle 90s → ReceiveTimeout → sends Empty to parent → parent's 30min timer resets →
Observer idle 90s → ReceiveTimeout → sends Empty to parent → parent's 30min timer resets →
... forever
Impact
- Zombie sessions: Every session that has ever run stays alive permanently
- GPU/inference load: Zombie sessions accumulate; when any receives new messages (e.g., Slack thread reply), the session is still warm and runs distillation LLM calls with growing transcripts
- Memory: Actor state for all sessions held in memory indefinitely
- Observed: Session ran 04:23→17:22+ (13hrs), 435 distillation cycles, 8 real LLM calls (up to 14,956 input tokens each), multiple
OutputStreamTerminated dead letter errors
Evidence from logs
# 435 distillation cycles, most empty
session_distillation_completed proposals=0 inputTokens=null outputTokens=null (×427)
# 8 real LLM calls that should not have happened (session should have been long dead)
session_distillation_completed proposals=1 inputTokens=1926 outputTokens=1259
session_distillation_completed proposals=3 inputTokens=5384 outputTokens=1808
session_distillation_completed proposals=0 inputTokens=10399 outputTokens=686
session_distillation_completed proposals=3 inputTokens=11687 outputTokens=1294
session_distillation_completed proposals=3 inputTokens=12089 outputTokens=1686
session_distillation_completed proposals=4 inputTokens=12577 outputTokens=2017
session_distillation_completed proposals=3 inputTokens=14956 outputTokens=1598
# Slack gateway passivated twice, but session actor stayed alive
Slack thread idle for 1 hour, passivating (05:25, 15:22)
Fix
When ReceiveTimeout fires on the observer and there's nothing to distill, silently return instead of messaging the parent. The explicit DistillMemories path (passivation handshake) must still always reply.
Command<ReceiveTimeout>(_ =>
{
if (_distilling || !_hasNewContent)
return; // silently skip — do not message parent
TriggerDistillation(Context.Parent);
});
A fix is on branch fix/console-output-leaks (will be moved to its own branch/PR).
Interaction with PR #417 and issue #423
PR #417 adds a Passivating state with DistillMemories → observer handshake, but has its own issues flagged in review:
_observerActor.Tell(new DistillMemories()) uses NoSender — the observer replies to Sender, so SessionDistillationCompleted goes to dead letters and the passivation path falls through to the timeout (review comment on line 1037)
- Passivation
SessionDistillationCompleted handler skips HandleDistillationResult — final distillation proposals are never routed through the gate/curation pipeline, and token usage is never emitted (review comment on line 1001)
- Buffered messages during passivation are lost —
_buffer is not persisted or replayed after stop (review comment on line 1015)
Issue #423 (formalize observer) should include this idle-keepalive fix as a prerequisite.
Refs
Summary
The
SessionMemoryObserverActorprevents its parentLlmSessionActorfrom ever passivating. Sessions stay alive indefinitely — we observed sessionD0AC6CKBK5K/1774412603.599179running for 13+ hours with 435 distillation cycles, the vast majority of which did nothing.Root cause
The observer uses
Context.SetReceiveTimeout(idleTimeout)(90s) to trigger periodic distillation. When there's no new content,TriggerDistillationsendsSessionDistillationCompleted.Emptyback to the parent:In Akka.NET, any received message (except
ReceiveTimeoutitself) resets the actor's receive timeout timer. The parent's 30-minute idle passivation timeout (Context.SetReceiveTimeout(_config.IdleTimeout)) gets reset every 90 seconds by these empty distillation results, so the session never passivates.The feedback loop
Impact
OutputStreamTerminateddead letter errorsEvidence from logs
Fix
When
ReceiveTimeoutfires on the observer and there's nothing to distill, silently return instead of messaging the parent. The explicitDistillMemoriespath (passivation handshake) must still always reply.A fix is on branch
fix/console-output-leaks(will be moved to its own branch/PR).Interaction with PR #417 and issue #423
PR #417 adds a
Passivatingstate withDistillMemories→ observer handshake, but has its own issues flagged in review:_observerActor.Tell(new DistillMemories())usesNoSender— the observer replies toSender, soSessionDistillationCompletedgoes to dead letters and the passivation path falls through to the timeout (review comment on line 1037)SessionDistillationCompletedhandler skipsHandleDistillationResult— final distillation proposals are never routed through the gate/curation pipeline, and token usage is never emitted (review comment on line 1001)_bufferis not persisted or replayed after stop (review comment on line 1015)Issue #423 (formalize observer) should include this idle-keepalive fix as a prerequisite.
Refs