Skip to content

bug: SessionMemoryObserverActor ReceiveTimeout keeps sessions alive forever #426

@Aaronontheweb

Description

@Aaronontheweb

Summary

The SessionMemoryObserverActor prevents its parent LlmSessionActor from ever passivating. Sessions stay alive indefinitely — we observed session D0AC6CKBK5K/1774412603.599179 running for 13+ hours with 435 distillation cycles, the vast majority of which did nothing.

Root cause

The observer uses Context.SetReceiveTimeout(idleTimeout) (90s) to trigger periodic distillation. When there's no new content, TriggerDistillation sends SessionDistillationCompleted.Empty back to the parent:

// SessionMemoryObserverActor.cs:145-149
if (!_hasNewContent)
{
    replyTo.Tell(SessionDistillationCompleted.Empty);  // ← this resets parent's idle timer
    return;
}

In Akka.NET, any received message (except ReceiveTimeout itself) resets the actor's receive timeout timer. The parent's 30-minute idle passivation timeout (Context.SetReceiveTimeout(_config.IdleTimeout)) gets reset every 90 seconds by these empty distillation results, so the session never passivates.

The feedback loop

Observer idle 90s → ReceiveTimeout → sends Empty to parent → parent's 30min timer resets →
Observer idle 90s → ReceiveTimeout → sends Empty to parent → parent's 30min timer resets →
... forever

Impact

  • Zombie sessions: Every session that has ever run stays alive permanently
  • GPU/inference load: Zombie sessions accumulate; when any receives new messages (e.g., Slack thread reply), the session is still warm and runs distillation LLM calls with growing transcripts
  • Memory: Actor state for all sessions held in memory indefinitely
  • Observed: Session ran 04:23→17:22+ (13hrs), 435 distillation cycles, 8 real LLM calls (up to 14,956 input tokens each), multiple OutputStreamTerminated dead letter errors

Evidence from logs

# 435 distillation cycles, most empty
session_distillation_completed proposals=0 inputTokens=null outputTokens=null  (×427)

# 8 real LLM calls that should not have happened (session should have been long dead)
session_distillation_completed proposals=1 inputTokens=1926  outputTokens=1259
session_distillation_completed proposals=3 inputTokens=5384  outputTokens=1808
session_distillation_completed proposals=0 inputTokens=10399 outputTokens=686
session_distillation_completed proposals=3 inputTokens=11687 outputTokens=1294
session_distillation_completed proposals=3 inputTokens=12089 outputTokens=1686
session_distillation_completed proposals=4 inputTokens=12577 outputTokens=2017
session_distillation_completed proposals=3 inputTokens=14956 outputTokens=1598

# Slack gateway passivated twice, but session actor stayed alive
Slack thread idle for 1 hour, passivating  (05:25, 15:22)

Fix

When ReceiveTimeout fires on the observer and there's nothing to distill, silently return instead of messaging the parent. The explicit DistillMemories path (passivation handshake) must still always reply.

Command<ReceiveTimeout>(_ =>
{
    if (_distilling || !_hasNewContent)
        return; // silently skip — do not message parent
    TriggerDistillation(Context.Parent);
});

A fix is on branch fix/console-output-leaks (will be moved to its own branch/PR).

Interaction with PR #417 and issue #423

PR #417 adds a Passivating state with DistillMemories → observer handshake, but has its own issues flagged in review:

  1. _observerActor.Tell(new DistillMemories()) uses NoSender — the observer replies to Sender, so SessionDistillationCompleted goes to dead letters and the passivation path falls through to the timeout (review comment on line 1037)
  2. Passivation SessionDistillationCompleted handler skips HandleDistillationResult — final distillation proposals are never routed through the gate/curation pipeline, and token usage is never emitted (review comment on line 1001)
  3. Buffered messages during passivation are lost_buffer is not persisted or replayed after stop (review comment on line 1015)

Issue #423 (formalize observer) should include this idle-keepalive fix as a prerequisite.

Refs

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions