Skip to content

Commit fee91fe

Browse files
jalehmanjalehman
authored
feature(context): extend plugin system to support custom context management (#22201)
* feat(context-engine): add ContextEngine interface and registry Introduce the pluggable ContextEngine abstraction that allows external plugins to register custom context management strategies. - ContextEngine interface with lifecycle methods: bootstrap, ingest, ingestBatch, afterTurn, assemble, compact, prepareSubagentSpawn, onSubagentEnded, dispose - Module-level singleton registry with registerContextEngine() and resolveContextEngine() (config-driven slot selection) - LegacyContextEngine: pass-through implementation wrapping existing compaction behavior for 100% backward compatibility - ensureContextEnginesInitialized() guard for safe one-time registration - 19 tests covering contract, registry, resolution, and legacy parity * feat(plugins): add context-engine slot and registerContextEngine API Wire the ContextEngine abstraction into the plugin system so external plugins can register context engines via the standard plugin API. - Add 'context-engine' to PluginKind union type - Add 'contextEngine' slot to PluginSlotsConfig (default: 'legacy') - Wire registerContextEngine() through OpenClawPluginApi - Export ContextEngine types from plugin-sdk for external consumers - Restore proper slot-based resolution in registry * feat(context-engine): wire ContextEngine into agent run lifecycle Integrate the ContextEngine abstraction into the core agent run path: - Resolve context engine once per run (reused across retries) - Bootstrap: hydrate canonical store from session file on first run - Assemble: route context assembly through pluggable engine - Auto-compaction guard: disable built-in auto-compaction when the engine declares ownsCompaction (prevents double-compaction) - AfterTurn: post-turn lifecycle hook for ingest + background compaction decisions - Overflow compaction: route through contextEngine.compact() - Dispose: clean up engine resources in finally block - Notify context engine on subagent lifecycle events Legacy engine: all lifecycle methods are pass-through/no-op, preserving 100% backward compatibility for users without a context engine plugin. * feat(plugins): add scoped subagent methods and gateway request scope Expose runtime.subagent.{run, waitForRun, getSession, deleteSession} so external plugins can spawn sub-agent sessions without raw gateway dispatch access. Uses AsyncLocalStorage request-scope bridge to dispatch internally via handleGatewayRequest with a synthetic operator client. Methods are only available during gateway request handling. - Symbol.for-backed global singleton for cross-module-reload safety - Fallback gateway context for non-WS dispatch paths (Telegram/WhatsApp) - Set gateway request scope for all handlers, not just plugin handlers - 3 staleness tests for fallback context hardening * feat(context-engine): route /compact and sessions.get through context engine Wire the /compact command and sessions.get handler through the pluggable ContextEngine interface. - Thread tokenBudget and force parameters to context engine compact - Route /compact through contextEngine.compact() when registered - Wire sessions.get as runtime alias for plugin subagent dispatch - Add .pebbles/ to .gitignore * style: format with oxfmt 0.33.0 Fix duplicate import (ControlUiRootState in server.impl.ts) and import ordering across all changed files. * fix: update extension test mocks for context-engine types Add missing subagent property to bluebubbles PluginRuntime mock. Add missing registerContextEngine to lobster OpenClawPluginApi mock. * fix(subagents): keep deferred delete cleanup retryable * style: format run attempt for CI * fix(rebase): remove duplicate embedded-run imports * test: add missing gateway context mock export * fix: pass resolved auth profile into afterTurn compaction Ensure the embedded runner forwards resolved auth profile context into legacy context-engine compaction params on the normal afterTurn path, matching overflow compaction behavior. This allows downstream LCM summarization to use the intended provider auth/profile consistently. Also fix strict TS typing in external-link token dedupe and align an attempt unit test reasoningLevel value with the current ReasoningLevel enum. Regeneration-Prompt: | We were debugging context-engine compaction where downstream summary calls were missing the right auth/profile context in normal afterTurn flow, while overflow compaction already propagated it. Preserve current behavior and keep changes additive: thread the resolved authProfileId through run -> attempt -> legacy compaction param builder without broad refactors. Add tests that prove the auth profile is included in afterTurn legacy params and that overflow compaction still passes it through run attempts. Keep existing APIs stable, and only adjust small type issues needed for strict compilation. * fix: remove duplicate imports from rebase * feat: add context-engine system prompt additions * fix(rebase): dedupe attempt import declarations * test: fix fetch mock typing in ollama autodiscovery * fix(test): add registerContextEngine to diffs extension mock APIs * test(windows): use path.delimiter in ios-team-id fixture PATH * test(cron): add model formatting and precedence edge case tests Covers: - Provider/model string splitting (whitespace, nested paths, empty segments) - Provider normalization (casing, aliases like bedrock→amazon-bedrock) - Anthropic model alias normalization (opus-4.5→claude-opus-4-5) - Precedence: job payload > session override > config default - Sequential runs with different providers (CI flake regression pattern) - forceNew session preserving stored model overrides - Whitespace/empty model string edge cases - Config model as string vs object format * test(cron): fix model formatting test config types * test(phone-control): add registerContextEngine to mock API * fix: re-export ChannelKind from config-reload-plan * fix: add subagent mock to plugin-runtime-mock test util * docs: add changelog fragment for context engine PR #22201
1 parent fa6c0e1 commit fee91fe

44 files changed

Lines changed: 2308 additions & 103 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ Docs: https://docs.openclaw.ai
2323
- Tools/Diffs guidance: restore a short system-prompt hint for enabled diffs while keeping the detailed instructions in the companion skill, so diffs usage guidance stays out of user-prompt space. (#36904) thanks @gumadeiras.
2424
- Telegram/ACP topic bindings: accept Telegram Mac Unicode dash option prefixes in `/acp spawn`, support Telegram topic thread binding (`--thread here|auto`), route bound-topic follow-ups to ACP sessions, add actionable Telegram approval buttons with prefixed approval-id resolution, and pin successful bind confirmations in-topic. (#36683) Thanks @huntharo.
2525
- Hooks/Compaction lifecycle: emit `session:compact:before` and `session:compact:after` internal events plus plugin compaction callbacks with session/count metadata, so automations can react to compaction runs consistently. (#16788) thanks @vincentkoc.
26+
- Agents/context engine plugin interface: add `ContextEngine` plugin slot with full lifecycle hooks (`bootstrap`, `ingest`, `assemble`, `compact`, `afterTurn`, `prepareSubagentSpawn`, `onSubagentEnded`), slot-based registry with config-driven resolution, `LegacyContextEngine` wrapper preserving existing compaction behavior, scoped subagent runtime for plugin runtimes via `AsyncLocalStorage`, and `sessions.get` gateway method. Enables plugins like `lossless-claw` to provide alternative context management strategies without modifying core compaction logic. Zero behavior change when no context engine plugin is configured. (#22201) thanks @jalehman.
2627
- CLI: make read-only SecretRef status flows degrade safely (#37023) thanks @joshavant.
2728

2829
### Breaking

‎extensions/diffs/index.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ describe("diffs plugin registration", () => {
3030
registerService() {},
3131
registerProvider() {},
3232
registerCommand() {},
33+
registerContextEngine() {},
3334
resolvePath(input: string) {
3435
return input;
3536
},
@@ -105,6 +106,7 @@ describe("diffs plugin registration", () => {
105106
registerService() {},
106107
registerProvider() {},
107108
registerCommand() {},
109+
registerContextEngine() {},
108110
resolvePath(input: string) {
109111
return input;
110112
},

‎extensions/diffs/src/tool.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -441,6 +441,7 @@ function createApi(): OpenClawPluginApi {
441441
registerService() {},
442442
registerProvider() {},
443443
registerCommand() {},
444+
registerContextEngine() {},
444445
resolvePath(input: string) {
445446
return input;
446447
},

‎extensions/lobster/src/lobster-tool.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ function fakeApi(overrides: Partial<OpenClawPluginApi> = {}): OpenClawPluginApi
4646
registerHook() {},
4747
registerHttpRoute() {},
4848
registerCommand() {},
49+
registerContextEngine() {},
4950
on() {},
5051
resolvePath: (p) => p,
5152
...overrides,

‎extensions/phone-control/index.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ function createApi(params: {
3939
registerCli() {},
4040
registerService() {},
4141
registerProvider() {},
42+
registerContextEngine() {},
4243
registerCommand: params.registerCommand,
4344
resolvePath(input: string) {
4445
return input;

‎extensions/test-utils/plugin-runtime-mock.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,13 @@ export function createPluginRuntimeMock(overrides: DeepPartial<PluginRuntime> =
242242
state: {
243243
resolveStateDir: vi.fn(() => "/tmp/openclaw"),
244244
},
245+
subagent: {
246+
run: vi.fn(),
247+
waitForRun: vi.fn(),
248+
getSessionMessages: vi.fn(),
249+
getSession: vi.fn(),
250+
deleteSession: vi.fn(),
251+
},
245252
};
246253

247254
return mergeDeep(base, overrides);

‎src/agents/pi-embedded-runner/compact.ts‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ import { resolveHeartbeatPrompt } from "../../auto-reply/heartbeat.js";
1111
import type { ReasoningLevel, ThinkLevel } from "../../auto-reply/thinking.js";
1212
import { resolveChannelCapabilities } from "../../config/channel-capabilities.js";
1313
import type { OpenClawConfig } from "../../config/config.js";
14+
import {
15+
ensureContextEnginesInitialized,
16+
resolveContextEngine,
17+
} from "../../context-engine/index.js";
1418
import { createInternalHookEvent, triggerInternalHook } from "../../hooks/internal-hooks.js";
1519
import { getMachineDisplayName } from "../../infra/machine-name.js";
1620
import { generateSecureToken } from "../../infra/secure-random.js";
@@ -29,8 +33,9 @@ import { resolveSessionAgentIds } from "../agent-scope.js";
2933
import type { ExecElevatedDefaults } from "../bash-tools.js";
3034
import { makeBootstrapWarn, resolveBootstrapContextForRun } from "../bootstrap-files.js";
3135
import { listChannelSupportedActions, resolveChannelMessageToolHints } from "../channel-tools.js";
36+
import { resolveContextWindowInfo } from "../context-window-guard.js";
3237
import { formatUserTime, resolveUserTimeFormat, resolveUserTimezone } from "../date-time.js";
33-
import { DEFAULT_MODEL, DEFAULT_PROVIDER } from "../defaults.js";
38+
import { DEFAULT_CONTEXT_TOKENS, DEFAULT_MODEL, DEFAULT_PROVIDER } from "../defaults.js";
3439
import { resolveOpenClawDocsPath } from "../docs-path.js";
3540
import { getApiKeyForModel, resolveModelAuthMode } from "../model-auth.js";
3641
import { ensureOpenClawModelsJson } from "../models-config.js";
@@ -115,6 +120,8 @@ export type CompactEmbeddedPiSessionParams = {
115120
reasoningLevel?: ReasoningLevel;
116121
bashElevated?: ExecElevatedDefaults;
117122
customInstructions?: string;
123+
tokenBudget?: number;
124+
force?: boolean;
118125
trigger?: "overflow" | "manual";
119126
diagId?: string;
120127
attempt?: number;
@@ -846,6 +853,49 @@ export async function compactEmbeddedPiSession(
846853
const enqueueGlobal =
847854
params.enqueue ?? ((task, opts) => enqueueCommandInLane(globalLane, task, opts));
848855
return enqueueCommandInLane(sessionLane, () =>
849-
enqueueGlobal(async () => compactEmbeddedPiSessionDirect(params)),
856+
enqueueGlobal(async () => {
857+
ensureContextEnginesInitialized();
858+
const contextEngine = await resolveContextEngine(params.config);
859+
try {
860+
// Resolve token budget from model context window so the context engine
861+
// knows the compaction target. The runner's afterTurn path passes this
862+
// automatically, but the /compact command path needs to compute it here.
863+
const ceProvider = (params.provider ?? DEFAULT_PROVIDER).trim() || DEFAULT_PROVIDER;
864+
const ceModelId = (params.model ?? DEFAULT_MODEL).trim() || DEFAULT_MODEL;
865+
const agentDir = params.agentDir ?? resolveOpenClawAgentDir();
866+
const { model: ceModel } = resolveModel(ceProvider, ceModelId, agentDir, params.config);
867+
const ceCtxInfo = resolveContextWindowInfo({
868+
cfg: params.config,
869+
provider: ceProvider,
870+
modelId: ceModelId,
871+
modelContextWindow: ceModel?.contextWindow,
872+
defaultTokens: DEFAULT_CONTEXT_TOKENS,
873+
});
874+
const result = await contextEngine.compact({
875+
sessionId: params.sessionId,
876+
sessionFile: params.sessionFile,
877+
tokenBudget: ceCtxInfo.tokens,
878+
customInstructions: params.customInstructions,
879+
force: params.trigger === "manual",
880+
legacyParams: params as Record<string, unknown>,
881+
});
882+
return {
883+
ok: result.ok,
884+
compacted: result.compacted,
885+
reason: result.reason,
886+
result: result.result
887+
? {
888+
summary: result.result.summary ?? "",
889+
firstKeptEntryId: result.result.firstKeptEntryId ?? "",
890+
tokensBefore: result.result.tokensBefore,
891+
tokensAfter: result.result.tokensAfter,
892+
details: result.result.details,
893+
}
894+
: undefined,
895+
};
896+
} finally {
897+
await contextEngine.dispose?.();
898+
}
899+
}),
850900
);
851901
}

‎src/agents/pi-embedded-runner/run.overflow-compaction.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,22 @@ describe("runEmbeddedPiAgent overflow compaction trigger routing", () => {
5454
);
5555
});
5656

57+
it("passes resolved auth profile into run attempts for context-engine afterTurn propagation", async () => {
58+
mockedRunEmbeddedAttempt.mockResolvedValueOnce(makeAttemptResult({ promptError: null }));
59+
60+
await runEmbeddedPiAgent({
61+
...overflowBaseRunParams,
62+
runId: "run-auth-profile-passthrough",
63+
});
64+
65+
expect(mockedRunEmbeddedAttempt).toHaveBeenCalledWith(
66+
expect.objectContaining({
67+
authProfileId: "test-profile",
68+
authProfileIdSource: "auto",
69+
}),
70+
);
71+
});
72+
5773
it("passes trigger=overflow when retrying compaction after context overflow", async () => {
5874
mockOverflowRetrySuccess({
5975
runEmbeddedAttempt: mockedRunEmbeddedAttempt,

‎src/agents/pi-embedded-runner/run.ts‎

Lines changed: 41 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
import { randomBytes } from "node:crypto";
22
import fs from "node:fs/promises";
33
import type { ThinkLevel } from "../../auto-reply/thinking.js";
4+
import {
5+
ensureContextEnginesInitialized,
6+
resolveContextEngine,
7+
} from "../../context-engine/index.js";
48
import { generateSecureToken } from "../../infra/secure-random.js";
59
import { getGlobalHookRunner } from "../../plugins/hook-runner-global.js";
610
import type { PluginHookBeforeAgentStartResult } from "../../plugins/types.js";
@@ -50,7 +54,6 @@ import {
5054
} from "../pi-embedded-helpers.js";
5155
import { derivePromptTokens, normalizeUsage, type UsageLike } from "../usage.js";
5256
import { redactRunIdentifier, resolveRunWorkspaceDir } from "../workspace-run.js";
53-
import { compactEmbeddedPiSessionDirect } from "./compact.js";
5457
import { resolveGlobalLane, resolveSessionLane } from "./lanes.js";
5558
import { log } from "./logger.js";
5659
import { resolveModel } from "./model.js";
@@ -737,6 +740,10 @@ export async function runEmbeddedPiAgent(
737740
agentDir,
738741
});
739742
};
743+
// Resolve the context engine once and reuse across retries to avoid
744+
// repeated initialization/connection overhead per attempt.
745+
ensureContextEnginesInitialized();
746+
const contextEngine = await resolveContextEngine(params.config);
740747
try {
741748
let authRetryPending = false;
742749
// Hoisted so the retry-limit error path can use the most recent API total.
@@ -806,13 +813,17 @@ export async function runEmbeddedPiAgent(
806813
workspaceDir: resolvedWorkspace,
807814
agentDir,
808815
config: params.config,
816+
contextEngine,
817+
contextTokenBudget: ctxInfo.tokens,
809818
skillsSnapshot: params.skillsSnapshot,
810819
prompt,
811820
images: params.images,
812821
disableTools: params.disableTools,
813822
provider,
814823
modelId,
815824
model,
825+
authProfileId: lastProfileId,
826+
authProfileIdSource: lockedProfileId ? "user" : "auto",
816827
authStorage,
817828
modelRegistry,
818829
agentId: workspaceResolution.agentId,
@@ -955,31 +966,36 @@ export async function runEmbeddedPiAgent(
955966
log.warn(
956967
`context overflow detected (attempt ${overflowCompactionAttempts}/${MAX_OVERFLOW_COMPACTION_ATTEMPTS}); attempting auto-compaction for ${provider}/${modelId}`,
957968
);
958-
const compactResult = await compactEmbeddedPiSessionDirect({
969+
const compactResult = await contextEngine.compact({
959970
sessionId: params.sessionId,
960-
sessionKey: params.sessionKey,
961-
messageChannel: params.messageChannel,
962-
messageProvider: params.messageProvider,
963-
agentAccountId: params.agentAccountId,
964-
authProfileId: lastProfileId,
965971
sessionFile: params.sessionFile,
966-
workspaceDir: resolvedWorkspace,
967-
agentDir,
968-
config: params.config,
969-
skillsSnapshot: params.skillsSnapshot,
970-
senderIsOwner: params.senderIsOwner,
971-
provider,
972-
model: modelId,
973-
runId: params.runId,
974-
thinkLevel,
975-
reasoningLevel: params.reasoningLevel,
976-
bashElevated: params.bashElevated,
977-
extraSystemPrompt: params.extraSystemPrompt,
978-
ownerNumbers: params.ownerNumbers,
979-
trigger: "overflow",
980-
diagId: overflowDiagId,
981-
attempt: overflowCompactionAttempts,
982-
maxAttempts: MAX_OVERFLOW_COMPACTION_ATTEMPTS,
972+
tokenBudget: ctxInfo.tokens,
973+
force: true,
974+
compactionTarget: "budget",
975+
legacyParams: {
976+
sessionKey: params.sessionKey,
977+
messageChannel: params.messageChannel,
978+
messageProvider: params.messageProvider,
979+
agentAccountId: params.agentAccountId,
980+
authProfileId: lastProfileId,
981+
workspaceDir: resolvedWorkspace,
982+
agentDir,
983+
config: params.config,
984+
skillsSnapshot: params.skillsSnapshot,
985+
senderIsOwner: params.senderIsOwner,
986+
provider,
987+
model: modelId,
988+
runId: params.runId,
989+
thinkLevel,
990+
reasoningLevel: params.reasoningLevel,
991+
bashElevated: params.bashElevated,
992+
extraSystemPrompt: params.extraSystemPrompt,
993+
ownerNumbers: params.ownerNumbers,
994+
trigger: "overflow",
995+
diagId: overflowDiagId,
996+
attempt: overflowCompactionAttempts,
997+
maxAttempts: MAX_OVERFLOW_COMPACTION_ATTEMPTS,
998+
},
983999
});
9841000
if (compactResult.compacted) {
9851001
autoCompactionCount += 1;
@@ -1412,6 +1428,7 @@ export async function runEmbeddedPiAgent(
14121428
};
14131429
}
14141430
} finally {
1431+
await contextEngine.dispose?.();
14151432
stopCopilotRefreshTimer();
14161433
process.chdir(prevCwd);
14171434
}

‎src/agents/pi-embedded-runner/run/attempt.test.ts‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
import { describe, expect, it, vi } from "vitest";
22
import type { OpenClawConfig } from "../../../config/config.js";
33
import {
4+
buildAfterTurnLegacyCompactionParams,
45
composeSystemPromptWithHookContext,
56
isOllamaCompatProvider,
7+
prependSystemPromptAddition,
68
resolveAttemptFsWorkspaceOnly,
79
resolveOllamaBaseUrlForRun,
810
resolveOllamaCompatNumCtxEnabled,
@@ -180,7 +182,6 @@ describe("resolveAttemptFsWorkspaceOnly", () => {
180182
).toBe(false);
181183
});
182184
});
183-
184185
describe("wrapStreamFnTrimToolCallNames", () => {
185186
function createFakeStream(params: { events: unknown[]; resultMessage: unknown }): {
186187
result: () => Promise<unknown>;
@@ -548,3 +549,54 @@ describe("decodeHtmlEntitiesInObject", () => {
548549
expect(decodeHtmlEntitiesInObject("&#x27;world&#x27;")).toBe("'world'");
549550
});
550551
});
552+
describe("prependSystemPromptAddition", () => {
553+
it("prepends context-engine addition to the system prompt", () => {
554+
const result = prependSystemPromptAddition({
555+
systemPrompt: "base system",
556+
systemPromptAddition: "extra behavior",
557+
});
558+
559+
expect(result).toBe("extra behavior\n\nbase system");
560+
});
561+
562+
it("returns the original system prompt when no addition is provided", () => {
563+
const result = prependSystemPromptAddition({
564+
systemPrompt: "base system",
565+
});
566+
567+
expect(result).toBe("base system");
568+
});
569+
});
570+
571+
describe("buildAfterTurnLegacyCompactionParams", () => {
572+
it("includes resolved auth profile fields for context-engine afterTurn compaction", () => {
573+
const legacy = buildAfterTurnLegacyCompactionParams({
574+
attempt: {
575+
sessionKey: "agent:main:session:abc",
576+
messageChannel: "slack",
577+
messageProvider: "slack",
578+
agentAccountId: "acct-1",
579+
authProfileId: "openai:p1",
580+
config: { plugins: { slots: { contextEngine: "lossless-claw" } } } as OpenClawConfig,
581+
skillsSnapshot: undefined,
582+
senderIsOwner: true,
583+
provider: "openai-codex",
584+
modelId: "gpt-5.3-codex",
585+
thinkLevel: "off",
586+
reasoningLevel: "on",
587+
extraSystemPrompt: "extra",
588+
ownerNumbers: ["+15555550123"],
589+
},
590+
workspaceDir: "/tmp/workspace",
591+
agentDir: "/tmp/agent",
592+
});
593+
594+
expect(legacy).toMatchObject({
595+
authProfileId: "openai:p1",
596+
provider: "openai-codex",
597+
model: "gpt-5.3-codex",
598+
workspaceDir: "/tmp/workspace",
599+
agentDir: "/tmp/agent",
600+
});
601+
});
602+
});

0 commit comments

Comments
 (0)