Skip to content

Commit 0bcb95e

Browse files
authored
Models: enforce source-managed SecretRef markers in models.json (#43759)
Merged via squash. Prepared head SHA: 4a065ef Co-authored-by: joshavant <830519+joshavant@users.noreply.github.com> Co-authored-by: joshavant <830519+joshavant@users.noreply.github.com> Reviewed-by: @joshavant
1 parent e8a162d commit 0bcb95e

10 files changed

Lines changed: 390 additions & 23 deletions

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Docs: https://docs.openclaw.ai
1111
- Security/proxy attachments: restore the shared media-store size cap for persisted browser proxy files so oversized payloads are rejected instead of overriding the intended 5 MB limit. (`GHSA-6rph-mmhp-h7h9`)(#43684) Thanks @tdjackey and @vincentkoc.
1212
- Security/host env: block inherited `GIT_EXEC_PATH` from sanitized host exec environments so Git helper resolution cannot be steered by host environment state. (`GHSA-jf5v-pqgw-gm5m`)(#43685) Thanks @zpbrent and @vincentkoc.
1313
- Security/session_status: enforce sandbox session-tree visibility and shared agent-to-agent access guards before reading or mutating target session state, so sandboxed subagents can no longer inspect parent session metadata or write parent model overrides via `session_status`. (`GHSA-wcxr-59v9-rxr8`)(#43754) Thanks @tdjackey and @vincentkoc.
14+
- Models/secrets: enforce source-managed SecretRef markers in generated `models.json` so runtime-resolved provider secrets are not persisted when runtime projection is skipped. (#43759) Thanks @joshavant.
1415

1516
### Changes
1617

docs/cli/agent.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,4 +25,5 @@ openclaw agent --agent ops --message "Generate report" --deliver --reply-channel
2525

2626
## Notes
2727

28-
- When this command triggers `models.json` regeneration, SecretRef-managed provider credentials are persisted as non-secret markers (for example env var names or `secretref-managed`), not resolved secret plaintext.
28+
- When this command triggers `models.json` regeneration, SecretRef-managed provider credentials are persisted as non-secret markers (for example env var names, `secretref-env:ENV_VAR_NAME`, or `secretref-managed`), not resolved secret plaintext.
29+
- Marker writes are source-authoritative: OpenClaw persists markers from the active source config snapshot, not from resolved runtime secret values.

docs/concepts/models.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -207,15 +207,17 @@ mode, pass `--yes` to accept defaults.
207207
## Models registry (`models.json`)
208208

209209
Custom providers in `models.providers` are written into `models.json` under the
210-
agent directory (default `~/.openclaw/agents/<agentId>/models.json`). This file
210+
agent directory (default `~/.openclaw/agents/<agentId>/agent/models.json`). This file
211211
is merged by default unless `models.mode` is set to `replace`.
212212

213213
Merge mode precedence for matching provider IDs:
214214

215215
- Non-empty `baseUrl` already present in the agent `models.json` wins.
216216
- Non-empty `apiKey` in the agent `models.json` wins only when that provider is not SecretRef-managed in current config/auth-profile context.
217217
- SecretRef-managed provider `apiKey` values are refreshed from source markers (`ENV_VAR_NAME` for env refs, `secretref-managed` for file/exec refs) instead of persisting resolved secrets.
218+
- SecretRef-managed provider header values are refreshed from source markers (`secretref-env:ENV_VAR_NAME` for env refs, `secretref-managed` for file/exec refs).
218219
- Empty or missing agent `apiKey`/`baseUrl` fall back to config `models.providers`.
219220
- Other provider fields are refreshed from config and normalized catalog data.
220221

221-
This marker-based persistence applies whenever OpenClaw regenerates `models.json`, including command-driven paths like `openclaw agent`.
222+
Marker persistence is source-authoritative: OpenClaw writes markers from the active source config snapshot (pre-resolution), not from resolved runtime secret values.
223+
This applies whenever OpenClaw regenerates `models.json`, including command-driven paths like `openclaw agent`.

docs/gateway/configuration-reference.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2014,9 +2014,11 @@ OpenClaw uses the pi-coding-agent model catalog. Add custom providers via `model
20142014
- Non-empty agent `models.json` `baseUrl` values win.
20152015
- Non-empty agent `apiKey` values win only when that provider is not SecretRef-managed in current config/auth-profile context.
20162016
- SecretRef-managed provider `apiKey` values are refreshed from source markers (`ENV_VAR_NAME` for env refs, `secretref-managed` for file/exec refs) instead of persisting resolved secrets.
2017+
- SecretRef-managed provider header values are refreshed from source markers (`secretref-env:ENV_VAR_NAME` for env refs, `secretref-managed` for file/exec refs).
20172018
- Empty or missing agent `apiKey`/`baseUrl` fall back to `models.providers` in config.
20182019
- Matching model `contextWindow`/`maxTokens` use the higher value between explicit config and implicit catalog values.
20192020
- Use `models.mode: "replace"` when you want config to fully rewrite `models.json`.
2021+
- Marker persistence is source-authoritative: markers are written from the active source config snapshot (pre-resolution), not from resolved runtime secret values.
20202022

20212023
### Provider field details
20222024

docs/reference/secretref-credential-surface.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ Notes:
101101
- Plan entries target `profiles.*.key` / `profiles.*.token` and write sibling refs (`keyRef` / `tokenRef`).
102102
- Auth-profile refs are included in runtime resolution and audit coverage.
103103
- For SecretRef-managed model providers, generated `agents/*/agent/models.json` entries persist non-secret markers (not resolved secret values) for `apiKey`/header surfaces.
104+
- Marker persistence is source-authoritative: OpenClaw writes markers from the active source config snapshot (pre-resolution), not from resolved runtime secret values.
104105
- For web search:
105106
- In explicit provider mode (`tools.web.search.provider` set), only the selected provider key is active.
106107
- In auto mode (`tools.web.search.provider` unset), only the first provider key that resolves by precedence is active.

src/agents/models-config.plan.ts

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import {
66
type ExistingProviderConfig,
77
} from "./models-config.merge.js";
88
import {
9+
enforceSourceManagedProviderSecrets,
910
normalizeProviders,
1011
resolveImplicitProviders,
1112
type ProviderConfig,
@@ -86,6 +87,7 @@ async function resolveProvidersForMode(params: {
8687

8788
export async function planOpenClawModelsJson(params: {
8889
cfg: OpenClawConfig;
90+
sourceConfigForSecrets?: OpenClawConfig;
8991
agentDir: string;
9092
env: NodeJS.ProcessEnv;
9193
existingRaw: string;
@@ -106,6 +108,8 @@ export async function planOpenClawModelsJson(params: {
106108
agentDir,
107109
env,
108110
secretDefaults: cfg.secrets?.defaults,
111+
sourceProviders: params.sourceConfigForSecrets?.models?.providers,
112+
sourceSecretDefaults: params.sourceConfigForSecrets?.secrets?.defaults,
109113
secretRefManagedProviders,
110114
}) ?? providers;
111115
const mergedProviders = await resolveProvidersForMode({
@@ -115,7 +119,14 @@ export async function planOpenClawModelsJson(params: {
115119
secretRefManagedProviders,
116120
explicitBaseUrlProviders: resolveExplicitBaseUrlProviders(cfg.models),
117121
});
118-
const nextContents = `${JSON.stringify({ providers: mergedProviders }, null, 2)}\n`;
122+
const secretEnforcedProviders =
123+
enforceSourceManagedProviderSecrets({
124+
providers: mergedProviders,
125+
sourceProviders: params.sourceConfigForSecrets?.models?.providers,
126+
sourceSecretDefaults: params.sourceConfigForSecrets?.secrets?.defaults,
127+
secretRefManagedProviders,
128+
}) ?? mergedProviders;
129+
const nextContents = `${JSON.stringify({ providers: secretEnforcedProviders }, null, 2)}\n`;
119130

120131
if (params.existingRaw === nextContents) {
121132
return { action: "noop" };

src/agents/models-config.providers.normalize-keys.test.ts

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,10 @@ import path from "node:path";
44
import { describe, expect, it } from "vitest";
55
import type { OpenClawConfig } from "../config/config.js";
66
import { NON_ENV_SECRETREF_MARKER } from "./model-auth-markers.js";
7-
import { normalizeProviders } from "./models-config.providers.js";
7+
import {
8+
enforceSourceManagedProviderSecrets,
9+
normalizeProviders,
10+
} from "./models-config.providers.js";
811

912
describe("normalizeProviders", () => {
1013
it("trims provider keys so image models remain discoverable for custom providers", async () => {
@@ -136,4 +139,38 @@ describe("normalizeProviders", () => {
136139
await fs.rm(agentDir, { recursive: true, force: true });
137140
}
138141
});
142+
143+
it("ignores non-object provider entries during source-managed enforcement", () => {
144+
const providers = {
145+
openai: null,
146+
moonshot: {
147+
baseUrl: "https://api.moonshot.ai/v1",
148+
api: "openai-completions",
149+
apiKey: "sk-runtime-moonshot", // pragma: allowlist secret
150+
models: [],
151+
},
152+
} as unknown as NonNullable<NonNullable<OpenClawConfig["models"]>["providers"]>;
153+
154+
const sourceProviders: NonNullable<NonNullable<OpenClawConfig["models"]>["providers"]> = {
155+
openai: {
156+
baseUrl: "https://api.openai.com/v1",
157+
api: "openai-completions",
158+
apiKey: { source: "env", provider: "default", id: "OPENAI_API_KEY" }, // pragma: allowlist secret
159+
models: [],
160+
},
161+
moonshot: {
162+
baseUrl: "https://api.moonshot.ai/v1",
163+
api: "openai-completions",
164+
apiKey: { source: "env", provider: "default", id: "MOONSHOT_API_KEY" }, // pragma: allowlist secret
165+
models: [],
166+
},
167+
};
168+
169+
const enforced = enforceSourceManagedProviderSecrets({
170+
providers,
171+
sourceProviders,
172+
});
173+
expect((enforced as Record<string, unknown>).openai).toBeNull();
174+
expect(enforced?.moonshot?.apiKey).toBe("MOONSHOT_API_KEY"); // pragma: allowlist secret
175+
});
139176
});

src/agents/models-config.providers.ts

Lines changed: 159 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import {
44
DEFAULT_COPILOT_API_BASE_URL,
55
resolveCopilotApiToken,
66
} from "../providers/github-copilot-token.js";
7+
import { isRecord } from "../utils.js";
78
import { normalizeOptionalSecretInput } from "../utils/normalize-secret-input.js";
89
import { ensureAuthProfileStore, listProfilesForProvider } from "./auth-profiles.js";
910
import { discoverBedrockModels } from "./bedrock-discovery.js";
@@ -70,6 +71,11 @@ export { resolveOllamaApiBase } from "./models-config.providers.discovery.js";
7071

7172
type ModelsConfig = NonNullable<OpenClawConfig["models"]>;
7273
export type ProviderConfig = NonNullable<ModelsConfig["providers"]>[string];
74+
type SecretDefaults = {
75+
env?: string;
76+
file?: string;
77+
exec?: string;
78+
};
7379

7480
const ENV_VAR_NAME_RE = /^[A-Z_][A-Z0-9_]*$/;
7581

@@ -97,13 +103,7 @@ function resolveAwsSdkApiKeyVarName(env: NodeJS.ProcessEnv = process.env): strin
97103

98104
function normalizeHeaderValues(params: {
99105
headers: ProviderConfig["headers"] | undefined;
100-
secretDefaults:
101-
| {
102-
env?: string;
103-
file?: string;
104-
exec?: string;
105-
}
106-
| undefined;
106+
secretDefaults: SecretDefaults | undefined;
107107
}): { headers: ProviderConfig["headers"] | undefined; mutated: boolean } {
108108
const { headers } = params;
109109
if (!headers) {
@@ -276,15 +276,155 @@ function normalizeAntigravityProvider(provider: ProviderConfig): ProviderConfig
276276
return normalizeProviderModels(provider, normalizeAntigravityModelId);
277277
}
278278

279+
function normalizeSourceProviderLookup(
280+
providers: ModelsConfig["providers"] | undefined,
281+
): Record<string, ProviderConfig> {
282+
if (!providers) {
283+
return {};
284+
}
285+
const out: Record<string, ProviderConfig> = {};
286+
for (const [key, provider] of Object.entries(providers)) {
287+
const normalizedKey = key.trim();
288+
if (!normalizedKey || !isRecord(provider)) {
289+
continue;
290+
}
291+
out[normalizedKey] = provider;
292+
}
293+
return out;
294+
}
295+
296+
function resolveSourceManagedApiKeyMarker(params: {
297+
sourceProvider: ProviderConfig | undefined;
298+
sourceSecretDefaults: SecretDefaults | undefined;
299+
}): string | undefined {
300+
const sourceApiKeyRef = resolveSecretInputRef({
301+
value: params.sourceProvider?.apiKey,
302+
defaults: params.sourceSecretDefaults,
303+
}).ref;
304+
if (!sourceApiKeyRef || !sourceApiKeyRef.id.trim()) {
305+
return undefined;
306+
}
307+
return sourceApiKeyRef.source === "env"
308+
? sourceApiKeyRef.id.trim()
309+
: resolveNonEnvSecretRefApiKeyMarker(sourceApiKeyRef.source);
310+
}
311+
312+
function resolveSourceManagedHeaderMarkers(params: {
313+
sourceProvider: ProviderConfig | undefined;
314+
sourceSecretDefaults: SecretDefaults | undefined;
315+
}): Record<string, string> {
316+
const sourceHeaders = isRecord(params.sourceProvider?.headers)
317+
? (params.sourceProvider.headers as Record<string, unknown>)
318+
: undefined;
319+
if (!sourceHeaders) {
320+
return {};
321+
}
322+
const markers: Record<string, string> = {};
323+
for (const [headerName, headerValue] of Object.entries(sourceHeaders)) {
324+
const sourceHeaderRef = resolveSecretInputRef({
325+
value: headerValue,
326+
defaults: params.sourceSecretDefaults,
327+
}).ref;
328+
if (!sourceHeaderRef || !sourceHeaderRef.id.trim()) {
329+
continue;
330+
}
331+
markers[headerName] =
332+
sourceHeaderRef.source === "env"
333+
? resolveEnvSecretRefHeaderValueMarker(sourceHeaderRef.id)
334+
: resolveNonEnvSecretRefHeaderValueMarker(sourceHeaderRef.source);
335+
}
336+
return markers;
337+
}
338+
339+
export function enforceSourceManagedProviderSecrets(params: {
340+
providers: ModelsConfig["providers"];
341+
sourceProviders: ModelsConfig["providers"] | undefined;
342+
sourceSecretDefaults?: SecretDefaults;
343+
secretRefManagedProviders?: Set<string>;
344+
}): ModelsConfig["providers"] {
345+
const { providers } = params;
346+
if (!providers) {
347+
return providers;
348+
}
349+
const sourceProvidersByKey = normalizeSourceProviderLookup(params.sourceProviders);
350+
if (Object.keys(sourceProvidersByKey).length === 0) {
351+
return providers;
352+
}
353+
354+
let nextProviders: Record<string, ProviderConfig> | null = null;
355+
for (const [providerKey, provider] of Object.entries(providers)) {
356+
if (!isRecord(provider)) {
357+
continue;
358+
}
359+
const sourceProvider = sourceProvidersByKey[providerKey.trim()];
360+
if (!sourceProvider) {
361+
continue;
362+
}
363+
let nextProvider = provider;
364+
let providerMutated = false;
365+
366+
const sourceApiKeyMarker = resolveSourceManagedApiKeyMarker({
367+
sourceProvider,
368+
sourceSecretDefaults: params.sourceSecretDefaults,
369+
});
370+
if (sourceApiKeyMarker) {
371+
params.secretRefManagedProviders?.add(providerKey.trim());
372+
if (nextProvider.apiKey !== sourceApiKeyMarker) {
373+
providerMutated = true;
374+
nextProvider = {
375+
...nextProvider,
376+
apiKey: sourceApiKeyMarker,
377+
};
378+
}
379+
}
380+
381+
const sourceHeaderMarkers = resolveSourceManagedHeaderMarkers({
382+
sourceProvider,
383+
sourceSecretDefaults: params.sourceSecretDefaults,
384+
});
385+
if (Object.keys(sourceHeaderMarkers).length > 0) {
386+
const currentHeaders = isRecord(nextProvider.headers)
387+
? (nextProvider.headers as Record<string, unknown>)
388+
: undefined;
389+
const nextHeaders = {
390+
...(currentHeaders as Record<string, NonNullable<ProviderConfig["headers"]>[string]>),
391+
};
392+
let headersMutated = !currentHeaders;
393+
for (const [headerName, marker] of Object.entries(sourceHeaderMarkers)) {
394+
if (nextHeaders[headerName] === marker) {
395+
continue;
396+
}
397+
headersMutated = true;
398+
nextHeaders[headerName] = marker;
399+
}
400+
if (headersMutated) {
401+
providerMutated = true;
402+
nextProvider = {
403+
...nextProvider,
404+
headers: nextHeaders,
405+
};
406+
}
407+
}
408+
409+
if (!providerMutated) {
410+
continue;
411+
}
412+
if (!nextProviders) {
413+
nextProviders = { ...providers };
414+
}
415+
nextProviders[providerKey] = nextProvider;
416+
}
417+
418+
return nextProviders ?? providers;
419+
}
420+
279421
export function normalizeProviders(params: {
280422
providers: ModelsConfig["providers"];
281423
agentDir: string;
282424
env?: NodeJS.ProcessEnv;
283-
secretDefaults?: {
284-
env?: string;
285-
file?: string;
286-
exec?: string;
287-
};
425+
secretDefaults?: SecretDefaults;
426+
sourceProviders?: ModelsConfig["providers"];
427+
sourceSecretDefaults?: SecretDefaults;
288428
secretRefManagedProviders?: Set<string>;
289429
}): ModelsConfig["providers"] {
290430
const { providers } = params;
@@ -434,7 +574,13 @@ export function normalizeProviders(params: {
434574
next[normalizedKey] = normalizedProvider;
435575
}
436576

437-
return mutated ? next : providers;
577+
const normalizedProviders = mutated ? next : providers;
578+
return enforceSourceManagedProviderSecrets({
579+
providers: normalizedProviders,
580+
sourceProviders: params.sourceProviders,
581+
sourceSecretDefaults: params.sourceSecretDefaults,
582+
secretRefManagedProviders: params.secretRefManagedProviders,
583+
});
438584
}
439585

440586
type ImplicitProviderParams = {

0 commit comments

Comments
 (0)