Skip to content

Commit 362cfce

Browse files
committed
fix(codex): route workspace memory through tools
1 parent fdbf3cf commit 362cfce

7 files changed

Lines changed: 304 additions & 28 deletions

File tree

docs/concepts/system-prompt.md

Lines changed: 17 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -183,18 +183,22 @@ in every user turn. Codex loads `AGENTS.md` through its own project-doc
183183
discovery. `SOUL.md`, `IDENTITY.md`, `TOOLS.md`, and `USER.md` are forwarded as
184184
Codex developer instructions. `HEARTBEAT.md` content is not injected; heartbeat
185185
turns get a collaboration-mode note pointing to the file when it exists and is
186-
non-empty. `MEMORY.md` and active `BOOTSTRAP.md` content keep the normal
187-
turn-context role for now.
186+
non-empty. `MEMORY.md` content is not pasted into every native Codex turn;
187+
when memory tools are available, Codex turns get a small workspace-memory note
188+
and should use `memory_search` or `memory_get` when durable memory is relevant.
189+
If tools are disabled or memory search is unavailable, `MEMORY.md` falls back to
190+
the normal bounded turn-context path. Active `BOOTSTRAP.md` content keeps the
191+
normal turn-context role for now.
188192

189193
On non-Codex harnesses, bootstrap files continue to be composed into the
190194
OpenClaw prompt according to their existing gates. `HEARTBEAT.md` is omitted on
191195
normal runs when heartbeats are disabled for the default agent or
192196
`agents.defaults.heartbeat.includeSystemPromptSection` is false. Keep injected
193-
files concise, especially `MEMORY.md`. `MEMORY.md` is intended to stay a curated
194-
long-term summary; detailed daily notes belong in `memory/*.md` where
197+
files concise, especially non-Codex `MEMORY.md`. `MEMORY.md` is intended to stay
198+
a curated long-term summary; detailed daily notes belong in `memory/*.md` where
195199
`memory_search` and `memory_get` can retrieve them on demand. Oversized
196-
`MEMORY.md` files increase prompt usage and can be partially injected because of
197-
the bootstrap file limits below.
200+
non-Codex `MEMORY.md` files increase prompt usage and can be partially injected
201+
because of the bootstrap file limits below.
198202

199203
<Note>
200204
`memory/*.md` daily files are **not** part of the normal bootstrap Project Context. On ordinary turns they are accessed on demand via the `memory_search` and `memory_get` tools, so they do not count against the context window unless the model explicitly reads them. Bare `/new` and `/reset` turns are the exception: the runtime can prepend recent daily memory as a one-shot startup-context block for that first turn.
@@ -209,11 +213,13 @@ occurs, OpenClaw can inject a concise system-prompt warning notice; control this
209213
default: `always`). Detailed raw/injected counts stay in diagnostics such as
210214
`/context`, `/status`, doctor, and logs.
211215

212-
For memory files, truncation is not data loss: the file remains intact on disk,
213-
but the model only sees the shortened injected copy until it reads or searches
214-
memory directly. If `MEMORY.md` is repeatedly truncated, distill it into a
215-
shorter durable summary and move detailed history into `memory/*.md`, or
216-
intentionally raise the bootstrap limits.
216+
For memory files, truncation is not data loss: the file remains intact on disk.
217+
On native Codex, `MEMORY.md` is read on demand through memory tools when
218+
available, with bounded prompt fallback when tools cannot run. On other
219+
harnesses, the model only sees the shortened injected copy until it reads or
220+
searches memory directly. If `MEMORY.md` is repeatedly truncated there, distill
221+
it into a shorter durable summary and move detailed history into `memory/*.md`,
222+
or intentionally raise the bootstrap limits.
217223

218224
Sub-agent sessions only inject `AGENTS.md` and `TOOLS.md` (other bootstrap files
219225
are filtered out to keep the sub-agent context small).

docs/plugins/codex-harness-reference.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -420,8 +420,13 @@ files. `SOUL.md`, `IDENTITY.md`, `TOOLS.md`, and `USER.md` are forwarded as
420420
OpenClaw Codex developer instructions because they define the active agent,
421421
available workspace guidance, and user profile. `HEARTBEAT.md` content is not
422422
injected; heartbeat turns get a collaboration-mode pointer to read the file when
423-
it exists and is non-empty. `BOOTSTRAP.md` and `MEMORY.md` when present are
424-
forwarded as OpenClaw turn input reference context.
423+
it exists and is non-empty. `MEMORY.md` content is not pasted into native Codex
424+
turn input when memory tools are available; when it exists, the harness adds a
425+
small workspace-memory pointer and Codex should use `memory_search` or
426+
`memory_get` when durable memory is relevant. If tools are disabled or memory
427+
search is unavailable, `MEMORY.md` uses the normal bounded turn-context path.
428+
`BOOTSTRAP.md` when present is forwarded as OpenClaw turn input reference
429+
context.
425430

426431
## Environment overrides
427432

docs/reference/token-use.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ OpenClaw assembles its own system prompt on every run. It includes:
1919
with optional per-agent override at
2020
`agents.list[].skillsLimits.maxSkillsPromptChars`.
2121
- Self-update instructions
22-
- Workspace + bootstrap files (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md` when new, plus `MEMORY.md` when present). Lowercase root `memory.md` is not injected; it is legacy repair input for `openclaw doctor --fix` when paired with `MEMORY.md`. Large files are truncated by `agents.defaults.bootstrapMaxChars` (default: 12000), and total bootstrap injection is capped by `agents.defaults.bootstrapTotalMaxChars` (default: 60000). `memory/*.md` daily files are not part of the normal bootstrap prompt; they remain on-demand via memory tools on ordinary turns, but reset/startup model runs can prepend a one-shot startup-context block with recent daily memory for that first turn. Bare chat `/new` and `/reset` commands are acknowledged without invoking the model. The startup prelude is controlled by `agents.defaults.startupContext`. Post-compaction AGENTS.md excerpts are separate and require explicit `agents.defaults.compaction.postCompactionSections` opt-in.
22+
- Workspace + bootstrap files (`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`, `BOOTSTRAP.md` when new, plus `MEMORY.md` when present). Native Codex turns do not paste raw `MEMORY.md` when memory tools are available; they include a small memory pointer and use memory tools on demand. If tools are disabled or memory search is unavailable, `MEMORY.md` uses the normal bounded turn-context path. Lowercase root `memory.md` is not injected; it is legacy repair input for `openclaw doctor --fix` when paired with `MEMORY.md`. Large injected files are truncated by `agents.defaults.bootstrapMaxChars` (default: 12000), and total bootstrap injection is capped by `agents.defaults.bootstrapTotalMaxChars` (default: 60000). `memory/*.md` daily files are not part of the normal bootstrap prompt; they remain on-demand via memory tools on ordinary turns, but reset/startup model runs can prepend a one-shot startup-context block with recent daily memory for that first turn. Bare chat `/new` and `/reset` commands are acknowledged without invoking the model. The startup prelude is controlled by `agents.defaults.startupContext`. Post-compaction AGENTS.md excerpts are separate and require explicit `agents.defaults.compaction.postCompactionSections` opt-in.
2323
- Time (UTC + user timezone)
2424
- Reply tags + heartbeat behavior
2525
- Runtime metadata (host/OS/model/thinking)

extensions/codex/src/app-server/attempt-context.ts

Lines changed: 117 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
import { createHash } from "node:crypto";
22
import path from "node:path";
33
import {
4+
buildBootstrapContextForFiles,
45
embeddedAgentLog,
5-
resolveBootstrapContextForRun,
6+
resolveBootstrapFilesForRun,
67
type AgentMessage,
78
type ContextEngineProjection,
89
type EmbeddedContextFile,
@@ -32,6 +33,8 @@ const CODEX_WORKSPACE_DEVELOPER_CONTEXT_BASENAMES = new Set([
3233
...CODEX_TURN_SCOPED_WORKSPACE_DEVELOPER_CONTEXT_BASENAMES,
3334
]);
3435
const CODEX_HEARTBEAT_CONTEXT_BASENAME = "heartbeat.md";
36+
const CODEX_MEMORY_CONTEXT_BASENAME = "memory.md";
37+
const CODEX_MEMORY_TOOL_NAMES = new Set(["memory_search", "memory_get"]);
3538
const CODEX_BOOTSTRAP_CONTEXT_ORDER = new Map<string, number>([
3639
["soul.md", 10],
3740
["identity.md", 20],
@@ -42,15 +45,20 @@ const CODEX_BOOTSTRAP_CONTEXT_ORDER = new Map<string, number>([
4245
["heartbeat.md", 70],
4346
]);
4447

45-
type CodexBootstrapContext = Awaited<ReturnType<typeof resolveBootstrapContextForRun>>;
46-
type CodexBootstrapFile = CodexBootstrapContext["bootstrapFiles"][number];
48+
type CodexBootstrapFile = Awaited<ReturnType<typeof resolveBootstrapFilesForRun>>[number];
49+
type CodexBootstrapContext = {
50+
bootstrapFiles: CodexBootstrapFile[];
51+
contextFiles: EmbeddedContextFile[];
52+
};
4753
export type CodexSystemPromptReport = NonNullable<EmbeddedRunAttemptResult["systemPromptReport"]>;
4854
type CodexToolReportEntry = CodexSystemPromptReport["tools"]["entries"][number];
4955
type CodexWorkspaceBootstrapContext = CodexBootstrapContext & {
5056
promptContextFiles?: EmbeddedContextFile[];
5157
developerInstructionFiles?: EmbeddedContextFile[];
5258
turnScopedDeveloperInstructionFiles?: EmbeddedContextFile[];
5359
heartbeatReferenceFiles?: EmbeddedContextFile[];
60+
memoryReferenceFiles?: EmbeddedContextFile[];
61+
memoryToolRouted?: boolean;
5462
promptContext?: string;
5563
developerInstructions?: string;
5664
turnScopedDeveloperInstructions?: string;
@@ -201,9 +209,10 @@ export async function buildCodexWorkspaceBootstrapContext(params: {
201209
effectiveWorkspace: string;
202210
sessionKey: string;
203211
sessionAgentId: string;
212+
memoryToolsAvailable: boolean;
204213
}): Promise<CodexWorkspaceBootstrapContext> {
205214
try {
206-
const bootstrapContext = await resolveBootstrapContextForRun({
215+
const bootstrapFiles = await resolveBootstrapFilesForRun({
207216
workspaceDir: params.resolvedWorkspace,
208217
config: params.params.config,
209218
sessionKey: params.sessionKey,
@@ -213,14 +222,34 @@ export async function buildCodexWorkspaceBootstrapContext(params: {
213222
contextMode: params.params.bootstrapContextMode,
214223
runKind: params.params.bootstrapContextRunKind,
215224
});
216-
const contextFiles = bootstrapContext.contextFiles.map((file) =>
225+
const memoryReferenceFiles = params.memoryToolsAvailable
226+
? selectCodexWorkspaceMemoryReferenceFiles(bootstrapFiles).map((file) =>
227+
remapCodexContextFilePath({
228+
file: toCodexEmbeddedContextFile(file),
229+
sourceWorkspaceDir: params.resolvedWorkspace,
230+
targetWorkspaceDir: params.effectiveWorkspace,
231+
}),
232+
)
233+
: [];
234+
const contextFiles = buildBootstrapContextForFiles(
235+
params.memoryToolsAvailable
236+
? bootstrapFiles.filter((file) => !isCodexMemoryBootstrapFile(file))
237+
: bootstrapFiles,
238+
{
239+
config: params.params.config,
240+
agentId: params.params.agentId ?? params.sessionAgentId,
241+
warn: (message) => embeddedAgentLog.warn(message),
242+
},
243+
).map((file) =>
217244
remapCodexContextFilePath({
218245
file,
219246
sourceWorkspaceDir: params.resolvedWorkspace,
220247
targetWorkspaceDir: params.effectiveWorkspace,
221248
}),
222249
);
223-
const promptContextFiles = selectCodexWorkspacePromptContextFiles(contextFiles);
250+
const promptContextFiles = selectCodexWorkspacePromptContextFiles(contextFiles, {
251+
excludeMemory: params.memoryToolsAvailable,
252+
});
224253
const developerInstructionFiles = shouldInjectCodexOpenClawPromptContext(params.params)
225254
? selectCodexWorkspaceInheritedDeveloperInstructionFiles(contextFiles)
226255
: [];
@@ -231,12 +260,14 @@ export async function buildCodexWorkspaceBootstrapContext(params: {
231260
: [];
232261
const heartbeatReferenceFiles = selectCodexWorkspaceHeartbeatReferenceFiles(contextFiles);
233262
return {
234-
...bootstrapContext,
263+
bootstrapFiles,
235264
contextFiles,
236265
promptContextFiles,
237266
developerInstructionFiles,
238267
turnScopedDeveloperInstructionFiles,
239268
heartbeatReferenceFiles,
269+
memoryReferenceFiles,
270+
memoryToolRouted: params.memoryToolsAvailable,
240271
promptContext: renderCodexWorkspaceBootstrapPromptContext(promptContextFiles),
241272
developerInstructions:
242273
renderCodexWorkspaceThreadDeveloperInstructions(developerInstructionFiles),
@@ -293,6 +324,7 @@ export function buildCodexSystemPromptReport(params: {
293324
...(params.workspaceBootstrapContext.developerInstructionFiles ?? []),
294325
...(params.workspaceBootstrapContext.turnScopedDeveloperInstructionFiles ?? []),
295326
],
327+
memoryToolRouted: params.workspaceBootstrapContext.memoryToolRouted === true,
296328
}),
297329
skills: {
298330
promptChars: skillsPrompt.length,
@@ -388,6 +420,7 @@ function buildCodexBootstrapInjectionStats(params: {
388420
bootstrapFiles: CodexBootstrapFile[];
389421
injectedFiles: EmbeddedContextFile[];
390422
developerInstructionFiles?: EmbeddedContextFile[];
423+
memoryToolRouted?: boolean;
391424
}): CodexSystemPromptReport["injectedWorkspaceFiles"] {
392425
const injectedIndex = indexCodexContextFileContent(params.injectedFiles);
393426
const developerInstructionIndex = indexCodexContextFileContent(
@@ -411,6 +444,12 @@ function buildCodexBootstrapInjectionStats(params: {
411444
} else if (baseName === CODEX_HEARTBEAT_CONTEXT_BASENAME) {
412445
injectedChars = 0;
413446
truncated = false;
447+
} else if (
448+
baseName === CODEX_MEMORY_CONTEXT_BASENAME &&
449+
params.memoryToolRouted === true
450+
) {
451+
injectedChars = 0;
452+
truncated = false;
414453
}
415454
}
416455
return {
@@ -479,6 +518,7 @@ export function buildCodexOpenClawPromptContext(params: {
479518
params: EmbeddedRunAttemptParams;
480519
skillsPrompt?: string;
481520
workspacePromptContext?: string;
521+
workspaceMemoryReference?: string;
482522
}): string | undefined {
483523
if (!shouldInjectCodexOpenClawPromptContext(params.params)) {
484524
return undefined;
@@ -487,6 +527,7 @@ export function buildCodexOpenClawPromptContext(params: {
487527
params.skillsPrompt?.trim()
488528
? ["## OpenClaw Skills", "", params.skillsPrompt.trim()].join("\n")
489529
: undefined,
530+
params.workspaceMemoryReference?.trim() ? params.workspaceMemoryReference.trim() : undefined,
490531
params.workspacePromptContext?.trim()
491532
? ["## OpenClaw Workspace Context", "", params.workspacePromptContext.trim()].join("\n")
492533
: undefined,
@@ -557,7 +598,7 @@ function splitLeadingCodexDeliveryHint(prompt: string): {
557598
function renderCodexWorkspaceBootstrapPromptContext(
558599
contextFiles: EmbeddedContextFile[],
559600
): string | undefined {
560-
const files = selectCodexWorkspacePromptContextFiles(contextFiles);
601+
const files = contextFiles;
561602
if (files.length === 0) {
562603
return undefined;
563604
}
@@ -577,7 +618,9 @@ function renderCodexWorkspaceBootstrapPromptContext(
577618

578619
function selectCodexWorkspacePromptContextFiles(
579620
contextFiles: EmbeddedContextFile[],
621+
options: { excludeMemory?: boolean } = {},
580622
): EmbeddedContextFile[] {
623+
const excludeMemory = options.excludeMemory ?? true;
581624
return contextFiles
582625
.filter((file) => {
583626
const baseName = getCodexContextFileBasename(file.path);
@@ -586,6 +629,7 @@ function selectCodexWorkspacePromptContextFiles(
586629
!CODEX_NATIVE_PROJECT_DOC_BASENAMES.has(baseName) &&
587630
!CODEX_WORKSPACE_DEVELOPER_CONTEXT_BASENAMES.has(baseName) &&
588631
baseName !== CODEX_HEARTBEAT_CONTEXT_BASENAME &&
632+
(!excludeMemory || baseName !== CODEX_MEMORY_CONTEXT_BASENAME) &&
589633
!isMissingCodexBootstrapContextFile(file)
590634
);
591635
})
@@ -696,10 +740,58 @@ function renderCodexWorkspaceHeartbeatReference(files: EmbeddedContextFile[]): s
696740
return lines.join("\n").trim();
697741
}
698742

743+
function selectCodexWorkspaceMemoryReferenceFiles(
744+
bootstrapFiles: CodexBootstrapFile[],
745+
): CodexBootstrapFile[] {
746+
return bootstrapFiles
747+
.filter((file) => {
748+
const baseName = getCodexBootstrapFileBasename(file);
749+
return (
750+
baseName === CODEX_MEMORY_CONTEXT_BASENAME &&
751+
!file.missing &&
752+
(file.content ?? "").trim().length > 0
753+
);
754+
})
755+
.toSorted(compareCodexBootstrapFiles);
756+
}
757+
758+
export function renderCodexWorkspaceMemoryReference(params: {
759+
files: EmbeddedContextFile[];
760+
}): string | undefined {
761+
if (params.files.length === 0) {
762+
return undefined;
763+
}
764+
const lines = [
765+
"## OpenClaw Workspace Memory",
766+
"",
767+
`MEMORY.md exists in the active agent workspace. OpenClaw does not paste its contents into native Codex turns; use ${Array.from(CODEX_MEMORY_TOOL_NAMES).join(" or ")} when durable memory is relevant and the tools are available.`,
768+
"",
769+
];
770+
for (const file of params.files) {
771+
lines.push(`- ${file.path}`);
772+
}
773+
return lines.join("\n").trim();
774+
}
775+
776+
export function hasCodexWorkspaceMemoryTools(tools: readonly { name: string }[]): boolean {
777+
return tools.some((tool) => normalizeCodexDynamicToolName(tool.name) === "memory_search");
778+
}
779+
699780
function isMissingCodexBootstrapContextFile(file: EmbeddedContextFile): boolean {
700781
return file.content.trimStart().startsWith("[MISSING] Expected at:");
701782
}
702783

784+
function isCodexMemoryBootstrapFile(file: CodexBootstrapFile): boolean {
785+
return getCodexBootstrapFileBasename(file) === CODEX_MEMORY_CONTEXT_BASENAME;
786+
}
787+
788+
function toCodexEmbeddedContextFile(file: CodexBootstrapFile): EmbeddedContextFile {
789+
return {
790+
path: readNonEmptyString(file.path) ?? readNonEmptyString(file.name) ?? "",
791+
content: file.content ?? "",
792+
};
793+
}
794+
703795
export function remapCodexContextFilePath(params: {
704796
file: EmbeddedContextFile;
705797
sourceWorkspaceDir: string;
@@ -744,6 +836,13 @@ function compareCodexContextFiles(left: EmbeddedContextFile, right: EmbeddedCont
744836
return leftPath.localeCompare(rightPath);
745837
}
746838

839+
function compareCodexBootstrapFiles(left: CodexBootstrapFile, right: CodexBootstrapFile): number {
840+
return compareCodexContextFiles(
841+
toCodexEmbeddedContextFile(left),
842+
toCodexEmbeddedContextFile(right),
843+
);
844+
}
845+
747846
function normalizeCodexContextFilePath(filePath: string): string {
748847
return filePath.trim().replaceAll("\\", "/").toLowerCase();
749848
}
@@ -756,6 +855,16 @@ function getCodexContextFileBasename(filePath: string): string {
756855
return normalizeCodexContextFilePath(filePath).split("/").pop() ?? "";
757856
}
758857

858+
function getCodexBootstrapFileBasename(file: CodexBootstrapFile): string {
859+
return getCodexContextFileBasename(
860+
readNonEmptyString(file.path) ?? readNonEmptyString(file.name) ?? "",
861+
);
862+
}
863+
864+
function normalizeCodexDynamicToolName(name: string): string {
865+
return name.trim().toLowerCase();
866+
}
867+
759868
function isNonEmptyString(value: unknown): value is string {
760869
return typeof value === "string" && value.length > 0;
761870
}

0 commit comments

Comments
 (0)