sessions: add optional session bifurcation by 100yenadmin · Pull Request #339 · Martian-Engineering/lossless-claw

100yenadmin · 2026-04-09T07:20:18Z

Post-v0.8.0 context

v0.8.0 already shipped:

fix: harden malformed FTS migration recovery #334 malformed-FTS migration recovery
doctor (stack 1/4): add read-only cleaner diagnostics #336 read-only doctor diagnostics
doctor (stack 2/4): add backup-first cleaner apply #337 backup-first doctor cleanup apply

This PR is now one of the two remaining long-session follow-ups, and it depends on the retrieval family abstraction in #338.

Summary

Add optional session bifurcation so very long-lived keyed sessions can rotate into archived physical segments while still behaving like one logical session to retrieval.

Why this still matters

Lossless Claw is intentionally lossless. That means “summarize harder” is not enough when a session runs for weeks or months and keeps accumulating raw history in one physical conversation row set.

Without segmentation, a forever session can become increasingly expensive to inspect, recover, and maintain even if retrieval remains logically correct.

This PR introduces a controlled storage split without asking users to manually create a new session every few days.

What this PR changes

adds sessionBifurcation config with env and plugin-config support
wires the config into openclaw.plugin.json
rotates eligible active conversations into archived segments when thresholds are exceeded
keeps the feature disabled by default
excludes cron and subagent sessions from rotation
serializes replay dedupe, bifurcation, and ingest in the same per-session queue
preserves family-scoped retrieval through sessions: add session-family retrieval scope #338 after rotation

How it works

flowchart TD
  A[afterTurn event] --> B[Per-session queue]
  B --> C[Replay dedupe]
  C --> D{Threshold exceeded?}
  D -- no --> E[Ingest into current active segment]
  D -- yes --> F[Archive current segment]
  F --> G[Create or reuse fresh active segment]
  G --> E
  E --> H[Normal compaction and retrieval continue]

Trigger model

This does not run on a wall-clock schedule.

It runs opportunistically on the write path:

during afterTurn
after replay dedupe
before new messages are ingested

Rotation happens only when the active keyed session exceeds a configured threshold:

message-count threshold
age threshold, gated by a minimum message floor so old-but-tiny sessions do not split gratuitously

User-facing scenarios

1. A forever main session keeps running for weeks

The session keeps its logical identity while storage rolls forward into new physical segments.

2. A low-traffic but old session

Age alone is not enough; minMessagesBeforeAgeSplit prevents unnecessary rotation of tiny sessions.

3. Cron and subagent lanes

These are intentionally excluded because they have different lifecycle expectations and should not silently segment under the main-session policy.

Design decisions and trade-offs

Opt-in and off by default.
Default main behavior remains unchanged.
Threshold-driven, not scheduler-driven.
The goal is to split when storage pressure justifies it, not to create arbitrary daily slices.
Queue with ingest.
Rotation is serialized with replay dedupe and ingest so concurrent afterTurn paths cannot race.
Rebased onto current main after v0.8.0.
The branch now reflects shipped doctor changes and current upstream naming.

Validation

pnpm exec vitest run test/config.test.ts test/engine.test.ts test/lcm-command.test.ts test/lcm-expand-query-tool.test.ts test/lcm-tools.test.ts test/index-complete-provider-config.test.ts

Copilot

Pull request overview

This PR introduces infrastructure for long-lived sessions by (1) expanding retrieval/tool scoping to a session “family” (multiple conversation segments), (2) adding opt-in automatic session bifurcation (archive + new active segment) based on thresholds, and (3) adding “doctor cleaners” diagnostics/apply flows for high-confidence junk cleanup.

Changes:

Add conversation-family-aware scoping (conversationIds) across grep/describe/expand-query and the underlying message/summary search layers.
Add sessionBifurcation config (env + plugin config parsing) and implement automatic session rotation during afterTurn.
Add /lcm doctor cleaners and /lcm doctor cleaners apply plus tests.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`src/db/config.ts`	Adds `sessionBifurcation` config parsing and defaults.
`src/engine.ts`	Implements automatic rotation (archive + create) during `afterTurn`.
`src/store/conversation-store.ts`	Adds `conversationIds` search support, family lookup, and segment stats.
`src/store/summary-store.ts`	Adds `conversationIds` search support across summary query paths.
`src/retrieval.ts`	Plumbs `conversationIds` through RetrievalEngine.grep.
`src/tools/lcm-conversation-scope.ts`	Resolves session scope to a family (`conversationIds`) when available.
`src/tools/lcm-grep-tool.ts`	Passes `conversationIds` and updates scope text for families.
`src/tools/lcm-describe-tool.ts`	Enforces family scope for describe, updates error message.
`src/tools/lcm-expand-query-tool.ts`	Supports family grep + source conversation selection changes.
`src/plugin/lcm-doctor-cleaners.ts`	New cleaner scan/apply implementation with backup-first deletion.
`src/plugin/lcm-command.ts`	Adds command parsing + output for doctor cleaners scan/apply.
`test/config.test.ts`	Adds config parsing tests for `sessionBifurcation`.
`test/engine.test.ts`	Adds bifurcation behavior tests (threshold + cron exclusion).
`test/lcm-tools.test.ts`	Adds tests for family-scoped grep and updated describe behavior.
`test/lcm-expand-query-tool.test.ts`	Adds test for picking a single source conversation from family scope.
`test/lcm-command.test.ts`	Adds tests for doctor cleaners scan + apply workflows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

100yenadmin · 2026-04-10T11:12:51Z

Post-v0.8.0 rebase is complete. This branch is now aligned with current main, includes the follow-up rebase fixes for session-family expand-query scoping and afterTurn bifurcation logging, has 0 unresolved review threads, and the refreshed test check is passing. Ready to merge after #338 when a maintainer with upstream merge permission is available.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-12T19:37:35Z

+function appendConversationScopeConstraint(params: {
+  where: string[];
+  args: Array<string | number>;
+  columnExpr: string;
+  conversationId?: number;
+  conversationIds?: number[];
+}): void {
+  const normalizedConversationIds = [...new Set(
+    (params.conversationIds ?? [])
+      .filter((value) => Number.isFinite(value))
+      .map((value) => Math.trunc(value)),
+  )];
+  if (normalizedConversationIds.length > 0) {
+    if (normalizedConversationIds.length === 1) {
+      params.where.push(`${params.columnExpr} = ?`);
+      params.args.push(normalizedConversationIds[0]!);
+      return;
+    }
+    params.where.push(
+      `${params.columnExpr} IN (${normalizedConversationIds.map(() => "?").join(", ")})`,
+    );
+    params.args.push(...normalizedConversationIds);
+    return;
+  }
+  if (params.conversationId != null) {
+    params.where.push(`${params.columnExpr} = ?`);
+    params.args.push(params.conversationId);
+  }
+}
+


appendConversationScopeConstraint is imported at the top of this module and then re-declared as a local function. In an ES module/TypeScript this is a duplicate identifier and will fail to compile. Remove the local declaration (and use the shared helper in src/store/conversation-scope.ts), or alias one of them so there’s only a single binding.

Suggested change

function appendConversationScopeConstraint(params: {

where: string[];

args: Array<string | number>;

columnExpr: string;

conversationId?: number;

conversationIds?: number[];

}): void {

const normalizedConversationIds = [...new Set(

(params.conversationIds ?? [])

.filter((value) => Number.isFinite(value))

.map((value) => Math.trunc(value)),

)];

if (normalizedConversationIds.length > 0) {

if (normalizedConversationIds.length === 1) {

params.where.push(`${params.columnExpr} = ?`);

params.args.push(normalizedConversationIds[0]!);

return;

}

params.where.push(

`${params.columnExpr} IN (${normalizedConversationIds.map(() => "?").join(", ")})`,

);

params.args.push(...normalizedConversationIds);

return;

}

if (params.conversationId != null) {

params.where.push(`${params.columnExpr} = ?`);

params.args.push(params.conversationId);

}

}

Copilot · 2026-04-12T19:37:35Z

+function appendConversationScopeConstraint(params: {
+  where: string[];
+  args: Array<string | number>;
+  columnExpr: string;
+  conversationId?: ConversationId;
+  conversationIds?: ConversationId[];
+}): void {
+  const normalizedConversationIds = [...new Set(
+    (params.conversationIds ?? [])
+      .filter((value) => Number.isFinite(value))
+      .map((value) => Math.trunc(value)),
+  )];
+  if (normalizedConversationIds.length > 0) {
+    if (normalizedConversationIds.length === 1) {
+      params.where.push(`${params.columnExpr} = ?`);
+      params.args.push(normalizedConversationIds[0]!);
+      return;
+    }
+    params.where.push(
+      `${params.columnExpr} IN (${normalizedConversationIds.map(() => "?").join(", ")})`,
+    );
+    params.args.push(...normalizedConversationIds);
+    return;
+  }
+
+  if (params.conversationId != null) {
+    params.where.push(`${params.columnExpr} = ?`);
+    params.args.push(params.conversationId);
+  }
+}
+


appendConversationScopeConstraint is imported from ./conversation-scope.js and also re-declared locally in this file. That duplicate binding will cause a compile-time error. Remove the local helper and use the shared appendConversationScopeConstraint implementation (or alias the import) to avoid the duplicate identifier.

Suggested change

function appendConversationScopeConstraint(params: {

where: string[];

args: Array<string | number>;

columnExpr: string;

conversationId?: ConversationId;

conversationIds?: ConversationId[];

}): void {

const normalizedConversationIds = [...new Set(

(params.conversationIds ?? [])

.filter((value) => Number.isFinite(value))

.map((value) => Math.trunc(value)),

)];

if (normalizedConversationIds.length > 0) {

if (normalizedConversationIds.length === 1) {

params.where.push(`${params.columnExpr} = ?`);

params.args.push(normalizedConversationIds[0]!);

return;

}

params.where.push(

`${params.columnExpr} IN (${normalizedConversationIds.map(() => "?").join(", ")})`,

);

params.args.push(...normalizedConversationIds);

return;

}

if (params.conversationId != null) {

params.where.push(`${params.columnExpr} = ?`);

params.args.push(params.conversationId);

}

}

Copilot · 2026-04-12T19:37:35Z

  // ── Circuit breaker for compaction auth failures ──
  private circuitBreakerStates = new Map<string, CircuitBreakerState>();
+  private bifurcationMessageCountCache = new Map<number, number>();



bifurcationMessageCountCache is populated for active conversations but is only cleared when bifurcation rotates a segment. When sessions end (/reset, session_end, deleted, etc.) the conversation is archived via archiveConversation(...), but the cache entry is never removed, so the map can grow unbounded over long runtimes. Clear the cache when archiving a conversation in lifecycle paths (e.g., in applySessionReplacement after archiving) and/or cap the cache size.

Copilot · 2026-04-12T19:37:35Z

  /** Dynamic step-band policy for incremental leaf chunk sizing. */
  dynamicLeafChunkTokens: DynamicLeafChunkTokensConfig;
+  /** Optional automatic rotation of long-running sessions into archived segments. */
+  sessionBifurcation: SessionBifurcationConfig;


sessionBifurcation was added as a required field on LcmConfig. There are still several LcmConfig object literals in the test suite that don’t define sessionBifurcation (e.g., test/expansion.test.ts, test/circuit-breaker.test.ts, test/session-operation-queues.test.ts), which will cause TypeScript type-check failures. Either update those configs to include a default sessionBifurcation block, or make the field optional in the type and normalize it in resolveLcmConfig.

Suggested change

sessionBifurcation: SessionBifurcationConfig;

sessionBifurcation?: SessionBifurcationConfig;

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 9, 2026 07:20

Copilot started reviewing on behalf of 100yenadmin April 9, 2026 07:20 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread src/engine.ts Outdated

Comment thread src/tools/lcm-expand-query-tool.ts Outdated

Comment thread src/plugin/lcm-command.ts Outdated

Comment thread src/plugin/lcm-doctor-cleaners.ts Outdated

Comment thread src/db/config.ts

Comment thread test/lcm-command.test.ts Outdated

100yenadmin force-pushed the codex/session-bifurcation branch from cdd09da to 666a7ad Compare April 9, 2026 07:52

This was referenced Apr 9, 2026

doctor (stack 1/4): add read-only cleaner diagnostics #336

Merged

sessions: add session-family retrieval scope #338

Merged

100yenadmin changed the title ~~[codex] Add optional session bifurcation~~ sessions (stack 4/4): add optional session bifurcation Apr 9, 2026

100yenadmin requested a review from Copilot April 9, 2026 08:07

Copilot started reviewing on behalf of 100yenadmin April 9, 2026 08:07 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread src/store/conversation-store.ts Outdated

Comment thread src/engine.ts Outdated

Comment thread src/db/config.ts

100yenadmin force-pushed the codex/session-bifurcation branch from 666a7ad to 72c098d Compare April 9, 2026 08:39

100yenadmin requested a review from Copilot April 9, 2026 08:42

Copilot started reviewing on behalf of 100yenadmin April 9, 2026 08:43 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread src/engine.ts Outdated

Comment thread openclaw.plugin.json

100yenadmin force-pushed the codex/session-bifurcation branch from 72c098d to e2a0fad Compare April 9, 2026 08:55

100yenadmin requested a review from Copilot April 9, 2026 08:57

Copilot started reviewing on behalf of 100yenadmin April 9, 2026 08:57 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread src/plugin/lcm-doctor-cleaners.ts

100yenadmin force-pushed the codex/session-bifurcation branch from e2a0fad to 934bf15 Compare April 9, 2026 09:22

100yenadmin requested a review from Copilot April 9, 2026 10:24

Copilot started reviewing on behalf of 100yenadmin April 9, 2026 10:24 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Comment thread src/plugin/lcm-doctor-cleaners.ts Outdated

Comment thread src/plugin/lcm-command.ts

100yenadmin force-pushed the codex/session-bifurcation branch from 934bf15 to 9845556 Compare April 9, 2026 13:04

Eva added 7 commits April 10, 2026 14:34

add doctor cleaner apply workflow

1020341

add doctor cleaner apply workflow

1063b1a

add session family retrieval scope

cf76f98

dedupe retrieval scope helpers

1f65313

tighten session-family expansion scope

4ef10d0

dedupe doctor helper imports in session scope branch

9ed8303

skip family fallback when summaries are absent

e90470a

Eva added 9 commits April 10, 2026 14:34

trim scope branch follow-up surface

c4a4f31

restore family-aware expand query scoping after rebase

3fef39e

add session family retrieval scope

5a01061

trim scope branch follow-up surface

35a0f55

add optional session bifurcation

f29863b

stabilize bifurcation and family fallback

77a56aa

optimize bifurcation thresholds and docs

531541b

refine bifurcation cache accounting and ui hints

8a68514

fix afterTurn logging after bifurcation rebase

b24cec3

100yenadmin force-pushed the codex/session-bifurcation branch from 9845556 to b24cec3 Compare April 10, 2026 07:40

100yenadmin changed the title ~~sessions (stack 4/4): add optional session bifurcation~~ sessions: add optional session bifurcation Apr 10, 2026

100yenadmin mentioned this pull request Apr 10, 2026

Track remaining long-session architecture follow-up after v0.8.0 #335

Open

5 tasks

100yenadmin requested a review from Copilot April 12, 2026 19:32

Copilot started reviewing on behalf of 100yenadmin April 12, 2026 19:32 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

100yenadmin requested a review from Copilot May 3, 2026 17:44

Copilot started reviewing on behalf of 100yenadmin May 3, 2026 17:45 View session

Copilot AI reviewed May 3, 2026

View reviewed changes

100yenadmin closed this May 6, 2026

	sessionBifurcation: SessionBifurcationConfig;
	sessionBifurcation?: SessionBifurcationConfig;

Conversation

100yenadmin commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-v0.8.0 context

Summary

Why this still matters

What this PR changes

How it works

Trigger model

User-facing scenarios

1. A forever main session keeps running for weeks

2. A low-traffic but old session

3. Cron and subagent lanes

Design decisions and trade-offs

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

100yenadmin commented Apr 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

100yenadmin commented Apr 9, 2026 •

edited

Loading