Skip to content

config: prevent raw redaction overlap from corrupting config round-trips#32174

Merged
steipete merged 3 commits intoopenclaw:mainfrom
bmendonca3:bm/config-raw-redaction-collision
Mar 2, 2026
Merged

config: prevent raw redaction overlap from corrupting config round-trips#32174
steipete merged 3 commits intoopenclaw:mainfrom
bmendonca3:bm/config-raw-redaction-collision

Conversation

@bmendonca3
Copy link

Summary

Describe the problem and fix in 2–5 bullets:

  • Problem: config.get raw redaction can over-redact unrelated non-sensitive values when a sensitive value appears elsewhere in the JSON5 text.
  • Why it matters: round-tripping raw config through config.apply can fail validation (for example, gateway.mode becomes __OPENCLAW_REDACTED__).
  • What changed: added a safety check after text redaction; if the redacted raw cannot restore back to the original config shape, we fall back to structured redaction output (JSON5.stringify(redactedParsed)), which only redacts sensitive paths.
  • What did NOT change (scope boundary): object/parsed/resolved redaction behavior, sentinel value, and restore semantics are unchanged.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

User-visible / Behavior Changes

  • Raw config payloads from config.get no longer corrupt unrelated non-sensitive fields when secret values overlap with common literals.
  • Config Raw edits that previously failed on apply due to sentinel leakage in non-sensitive fields now round-trip successfully.

Security Impact (required)

  • New permissions/capabilities? (No)
  • Secrets/tokens handling changed? (Yes)
  • New/changed network calls? (No)
  • Command/tool execution surface changed? (No)
  • Data access scope changed? (No)
  • If any Yes, explain risk + mitigation:
    • Redaction output can now fall back to structured JSON5 rendering for raw responses when unsafe overlap is detected. This is a fail-closed behavior: it avoids leaking or corrupting values by preferring deterministic path-based redaction.

Repro + Verification

Environment

  • OS: macOS
  • Runtime/container: Node 22 + Vitest
  • Model/provider: N/A
  • Integration/channel (if any): Gateway config methods
  • Relevant config (redacted):
    • gateway.mode = "local"
    • gateway.auth.password = "local"

Steps

  1. Build snapshot with:
    • non-sensitive gateway.mode: "local"
    • sensitive gateway.auth.password: "local"
  2. Call redactConfigSnapshot(...) with schema hints.
  3. Parse returned raw and run restoreRedactedValues(...) + config validation.

Expected

  • Only gateway.auth.password is redacted.
  • Round-trip restore returns valid config with gateway.mode === "local".

Actual

  • Before fix, raw text replacement redacted both fields and turned gateway.mode into sentinel, causing invalid config on apply.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios:
    • Local repro script showed gateway.mode was incorrectly replaced in raw redaction before fallback logic.
    • Added regression test keeps non-sensitive raw fields intact when secret values overlap.
    • Ran pnpm vitest run src/config/redact-snapshot.test.ts --config vitest.unit.config.ts.
    • Ran pnpm check.
  • Edge cases checked:
    • Existing raw redaction test (redacts secrets in raw field via text-based redaction) still passes.
    • Full redact-snapshot suite passes.
  • What you did not verify:
    • Full browser Control UI manual flow was not exercised in this PR.

Compatibility / Migration

  • Backward compatible? (Yes)
  • Config/env changes? (No)
  • Migration needed? (No)
  • If yes, exact upgrade steps:

Failure Recovery (if this breaks)

  • How to disable/revert this change quickly:
    • Revert this PR commit.
  • Files/config to restore:
    • src/config/redact-snapshot.ts
    • src/config/redact-snapshot.test.ts
  • Known bad symptoms reviewers should watch for:
    • Unexpected raw formatting differences in config.get output where fallback redaction is triggered.

Risks and Mitigations

List only real risks for this PR. Add/remove entries as needed. If none, write None.

  • Risk:
    • Fallback rendering (JSON5.stringify) may not preserve original comments/formatting in affected raw snapshots.
    • Mitigation:
      • Fallback triggers only when overlap redaction is unsafe; parsed/config/resolved responses remain correct and round-trip-safe.

@aisle-research-bot
Copy link

aisle-research-bot bot commented Mar 2, 2026

🔒 Aisle Security Analysis

We found 2 potential security issue(s) in this PR:

# Severity Title
1 🟡 Medium Under-redaction of sensitive objects due to overly permissive SecretRef detection
2 🟡 Medium Prototype pollution via restoreRedactedValues assigning attacker-controlled keys ("proto"/"constructor"/"prototype")

1. 🟡 Under-redaction of sensitive objects due to overly permissive SecretRef detection

Property Value
Severity Medium
CWE CWE-200
Location src/config/redact-snapshot.ts:27-31

Description

src/config/redact-snapshot.ts introduces a new isSecretRefShape() type guard that treats any object with string source and string id as a SecretRef. This is weaker than the actual SecretRef schema/type in the codebase and can cause partial redaction of non‑SecretRef objects.

Why this can leak secrets:

  • Real SecretRef is defined as exactly { source: "env"|"file"|"exec", provider: string, id: string } and is validated with strict schemas (.strict()) (see src/config/types.secrets.ts and src/config/zod-schema.core.ts).
  • However, isSecretRefShape() accepts objects with any source string and no provider requirement, and it does not reject extra keys.
  • When a path is marked sensitive===true and the value is an object, redaction now does:
    • If isSecretRefShape(value) → redact only id (preserve the rest)
    • Else → collect all nested strings and replace the entire object with the sentinel

This creates a regression for sensitive fields that allow arbitrary objects. For example, channels.googlechat.serviceAccount is marked sensitive and allows an arbitrary object (z.record(z.string(), z.unknown())) in addition to SecretRef. If a user stores a service account-like object that happens to include top-level source and id keys plus other secret material (e.g., private_key), the new logic will:

  • redact only id
  • keep other properties intact in the structured response
  • and (because collectSensitiveStrings() also short-circuits on the same shape) fail to collect/replace those nested secret strings in result.raw

Vulnerable code:

function isSecretRefShape(value: Record<string, unknown>) {
  return typeof value.source === "string" && typeof value.id === "string";
}// ...
if (hints[candidate]?.sensitive === true && !Array.isArray(value)) {
  const objectValue = value as Record<string, unknown>;
  if (isSecretRefShape(objectValue)) {
    result[key] = redactSecretRef(objectValue, values); // only redacts id
  } else {
    collectSensitiveStrings(objectValue, values);
    result[key] = REDACTED_SENTINEL;
  }
}

This is an under-redaction / sensitive data exposure risk whenever a sensitive object field can legitimately contain non-SecretRef objects with source and id keys (even coincidentally).

Recommendation

Tighten SecretRef detection to match the actual SecretRef contract, and avoid short-circuiting on lookalike objects.

Recommended changes:

  1. Replace isSecretRefShape() with the existing strict guard from src/config/types.secrets.ts (or equivalent strict logic):
  • require source to be one of env|file|exec
  • require non-empty provider
  • require no extra keys (or explicitly handle extras)
  1. If an object has source/id but is not a strict SecretRef (e.g., missing provider or has extra keys), treat it as a normal sensitive object:
  • recurse with collectSensitiveStrings()
  • redact the entire object (REDACTED_SENTINEL)

Example hardening:

import { isSecretRef, type SecretRef } from "./types.secrets.js";

function isStrictSecretRef(value: unknown): value is SecretRef {
  return isSecretRef(value);
}// redactObjectWithLookup:
if (hints[candidate]?.sensitive === true && !Array.isArray(value)) {
  if (isStrictSecretRef(value)) {
    result[key] = redactSecretRef(value, values);
  } else {
    collectSensitiveStrings(value, values);
    result[key] = REDACTED_SENTINEL;
  }
}// collectSensitiveStrings: optionally keep special casing only for strict SecretRef
if (isStrictSecretRef(obj)) {
  if (!isEnvVarPlaceholder(obj.id)) values.push(obj.id);
  return;
}

This prevents lookalike objects from being partially redacted and ensures result.raw redaction also covers all nested secret strings.


2. 🟡 Prototype pollution via restoreRedactedValues assigning attacker-controlled keys ("proto"/"constructor"/"prototype")

Property Value
Severity Medium
CWE CWE-1321
Location src/config/redact-snapshot.ts:643-667

Description

restoreRedactedValues*() deep-walks an attacker-controlled object (e.g. parsed from JSON5 submitted to config.set/config.apply/config.patch, and now also parsed from snapshot.raw via shouldFallbackToStructuredRawRedaction) and copies keys into a fresh {} result object without blocking prototype keys.

Because result is a normal object, assigning to result["__proto__"] will invoke the __proto__ setter and mutate the prototype of result (prototype pollution on the returned/restored config object). This can introduce inherited properties that downstream code may read via property access or in checks, leading to unexpected behavior and potential validation/logic bypasses.

Vulnerable code:

const result: Record<string, unknown> = {};
for (const [key, value] of Object.entries(incoming as Record<string, unknown>)) {
  result[key] = value; // key may be "__proto__"// ...
}

New/expanded exposure in this diff:

  • shouldFallbackToStructuredRawRedaction() now does JSON5.parse(redactedRaw) and passes the parsed object to restoreRedactedValues(...), increasing the chance that a crafted config containing prototype keys triggers this behavior during config.get snapshot redaction as well.

Impact depends on where the restored object flows next, but the prototype mutation is concrete and avoidable.

Recommendation

Block prototype keys (at minimum __proto__, constructor, prototype) anywhere you copy/merge untrusted objects.

A minimal fix is to reuse the project’s existing helper (isBlockedObjectKey) and skip these keys in both restore workers:

import { isBlockedObjectKey } from "./prototype-keys.js";

const result: Record<string, unknown> = {};
for (const [key, value] of Object.entries(incoming as Record<string, unknown>)) {
  if (isBlockedObjectKey(key)) continue;// existing logic...
  result[key] = value;
}

Optionally, also create accumulator objects with Object.create(null) to remove magic __proto__ behavior entirely:

const result: Record<string, unknown> = Object.create(null);

Apply the same guard anywhere else in this file that copies arbitrary keys into {} (e.g., redaction walkers), for defense in depth.


Analyzed PR: #32174 at commit 3cc83c8

Last updated on: 2026-03-02T23:51:20Z

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4affcf6bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

redactedRaw &&
shouldFallbackToStructuredRawRedaction({
redactedRaw,
originalConfig: snapshot.config,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compare raw-fallback round-trip against parsed source

The fallback safety check restores redactedRaw against snapshot.config, but snapshot.config is normalized with runtime defaults while raw/parsed only represent user-authored fields (see src/config/io.ts where defaults are applied before snapshot.config is set). In common configs that omit defaults, this makes JSON.stringify(restored.result) !== JSON.stringify(snapshot.config) true even when text redaction was safe, so config.get falls back to JSON5.stringify(...) broadly and drops original comments/formatting. This check should be validated against the same shape as raw input (for example snapshot.parsed or pre-default resolved config) so fallback only triggers on actual overlap corruption.

Useful? React with 👍 / 👎.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced. The fallback comparator was using snapshot.config, which can include defaults not present in the parsed/raw shape, so safe text-redacted raw could be rewritten via structured fallback.

Applied on head 89a487c3ec:

  • fallback comparison now restores and compares against snapshot.parsed ?? snapshot.config.
  • added regression: keeps text redaction raw when parsed/config shapes differ by defaults to prove comments/formatting are preserved.

Verification:
pnpm -s vitest run src/config/redact-snapshot.test.ts

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 2, 2026

Greptile Summary

This PR fixes a raw-redaction overlap bug where a sensitive string value (e.g. "local" as a password) appeared elsewhere as a non-sensitive field value (e.g. gateway.mode: "local"), causing naïve replaceAll to corrupt the non-sensitive field and break config round-trips. The fix adds a two-layer defence: (1) restrict sensitive-value collection for SecretRef objects to only the id field (not structural fields like source/provider), and (2) after text-based redaction, probe the result via a suppressed restoreRedactedValues + isDeepStrictEqual round-trip; if the probe fails, fall back to JSON5.stringify of the structured (path-based) redacted config.

Key changes and observations:

  • isSecretRefShape is a loose duck-type guard requiring only source: string and id: string. Any object at an explicitly sensitive: true schema path that happens to carry those two fields will have only id redacted — other sensitive fields on the same object would be silently omitted from both text and structured redaction. A stricter guard (e.g. verifying no unrecognised string fields exist) would limit this to genuine SecretRef values (see inline comment).
  • The fallback comparison correctly uses isDeepStrictEqual from node:util (key-order-insensitive), addressing a potential concern with JSON.stringify-based comparison.
  • The suppressRestoreWarnings module-level mutable flag is safe for synchronous code but is a code smell; all callers are currently synchronous so there is no real concurrency risk.
  • When the fallback fires, original JSON5 comments and formatting are lost; this is acknowledged in the PR as an acceptable trade-off.

Confidence Score: 3/5

  • Safe to merge for the described overlap scenario, but the loose isSecretRefShape duck-type guard introduces a theoretical under-redaction path for non-SecretRef objects at sensitive schema paths that carry additional sensitive fields.
  • The core fix (round-trip fallback probe with isDeepStrictEqual) is sound and well-tested. The SecretRef-scoped collection correctly prevents over-redaction of structural fields. However, isSecretRefShape only checks for two specific string fields and could silently under-redact other sensitive data on objects that happen to share that shape — a tightened guard would make the heuristic unambiguous. No regressions in existing test coverage were introduced.
  • src/config/redact-snapshot.ts — the isSecretRefShape guard at lines 27-31 and its usages in collectSensitiveStrings and redactObjectWithLookup need attention.

Last reviewed commit: 3cc83c8

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

if (!restored.ok) {
return true;
}
return JSON.stringify(restored.result) !== JSON.stringify(params.originalConfig);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON.stringify comparison is key-order-sensitive — may trigger spurious fallbacks

Zod reconstructs objects in schema-definition key order, not necessarily the order in which keys appear in the raw JSON5 file. After restoreRedactedValues iterates over parsed (whose keys follow the raw file's order), the resulting object's key sequence may differ from params.originalConfig (which uses Zod's schema order). A simple JSON.stringify comparison will then return true even when the values are semantically identical, causing the fallback to fire for any config file where the user has keys in a different order than the schema — silently stripping comments and original formatting.

Consider a stable comparison instead, for example sorting keys before stringifying:

function stableStringify(value: unknown): string {
  return JSON.stringify(value, (_key, val) =>
    val && typeof val === "object" && !Array.isArray(val)
      ? Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b)))
      : val,
  );
}
// …
return stableStringify(restored.result) !== stableStringify(params.originalConfig);

This keeps the fail-closed safety guarantee while avoiding unnecessary fallbacks caused purely by key ordering.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/config/redact-snapshot.ts
Line: 309

Comment:
**`JSON.stringify` comparison is key-order-sensitive — may trigger spurious fallbacks**

Zod reconstructs objects in schema-definition key order, not necessarily the order in which keys appear in the raw JSON5 file. After `restoreRedactedValues` iterates over `parsed` (whose keys follow the raw file's order), the resulting object's key sequence may differ from `params.originalConfig` (which uses Zod's schema order). A simple `JSON.stringify` comparison will then return `true` even when the values are semantically identical, causing the fallback to fire for any config file where the user has keys in a different order than the schema — silently stripping comments and original formatting.

Consider a stable comparison instead, for example sorting keys before stringifying:

```ts
function stableStringify(value: unknown): string {
  return JSON.stringify(value, (_key, val) =>
    val && typeof val === "object" && !Array.isArray(val)
      ? Object.fromEntries(Object.entries(val).sort(([a], [b]) => a.localeCompare(b)))
      : val,
  );
}
//
return stableStringify(restored.result) !== stableStringify(params.originalConfig);
```

This keeps the fail-closed safety guarantee while avoiding unnecessary fallbacks caused purely by key ordering.

How can I resolve this? If you propose a fix, please make it concise.

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47a799b70f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +208 to +209
if (isSecretRefShape(objectValue)) {
result[key] = redactSecretRef(objectValue, values);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restrict SecretRef partial redaction to real SecretRef values

The new isSecretRefShape branch at sensitive object paths redacts only id and preserves the rest of the object, but the shape check is just source + id, so any sensitive object that happens to contain those keys is treated as a SecretRef. In particular, channels.googlechat.serviceAccount is sensitive and schema-valid as an arbitrary record (src/config/zod-schema.providers-core.ts:688), so an object like {source, id, private_key} would leak private_key in both config and raw outputs instead of being fully redacted as before.

Useful? React with 👍 / 👎.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. The prior source+id shape check was too broad for sensitive object paths and could preserve unintended fields on non-SecretRef objects.

Applied on head 89a487c3ec:

  • SecretRef partial-redaction now uses strict isSecretRef(...) validation.
  • non-SecretRef sensitive objects are fully redacted as before.
  • added regression: fully redacts non-SecretRef sensitive objects that include source/id keys.

Verification:
pnpm -s vitest run src/config/redact-snapshot.test.ts

@steipete steipete force-pushed the bm/config-raw-redaction-collision branch from 47a799b to 3cc83c8 Compare March 2, 2026 22:14
@steipete steipete merged commit bd4a082 into openclaw:main Mar 2, 2026
8 of 9 checks passed
@steipete
Copy link
Contributor

steipete commented Mar 2, 2026

Landed via temp rebase onto main.

  • Gate: pnpm -s vitest run src/config/redact-snapshot.test.ts
  • Land commit: 3cc83c8
  • Merge commit: bd4a082

Thanks @bmendonca3!

@chatgpt-codex-connector
Copy link

Codex Review: Didn't find any major issues. You're on a roll.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bmendonca3
Copy link
Author

@greptile review latest commit, please.

Comment on lines +27 to +31
function isSecretRefShape(
value: Record<string, unknown>,
): value is Record<string, unknown> & { source: string; id: string } {
return typeof value.source === "string" && typeof value.id === "string";
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isSecretRefShape duck-typing can silently under-redact non-SecretRef objects

isSecretRefShape only requires two fields (source: string, id: string) to be present. Any object at an explicitly sensitive: true schema path that happens to carry those two string fields will be treated as a SecretRef, and only id will be collected/replaced. Any additional sensitive fields on that object (e.g. a co-located apiKey, password, or token field) will be silently skipped by both redactSecretRef and the early-return branch in collectSensitiveStrings.

The fallback safety check (shouldFallbackToStructuredRawRedaction) does not help here: when the structured-redaction path is taken via redactSecretRef, the extra fields are already dropped from values before the text replacement, so they survive into the redactedParsed/redactedConfig objects that the fallback stringifies.

A narrower guard — for example also requiring that the object has no string fields other than source, provider, and id — would limit this to genuinely SecretRef-shaped values:

function isSecretRefShape(
  value: Record<string, unknown>,
): value is Record<string, unknown> & { source: string; id: string } {
  const NON_SENSITIVE_SECRETREF_KEYS = new Set(["source", "provider", "id"]);
  const keys = Object.keys(value);
  return (
    typeof value.source === "string" &&
    typeof value.id === "string" &&
    keys.every((k) => NON_SENSITIVE_SECRETREF_KEYS.has(k))
  );
}

This ensures the heuristic fires only when the object is structurally exactly a SecretRef, not any wider object that happens to contain source and id.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/config/redact-snapshot.ts
Line: 27-31

Comment:
**`isSecretRefShape` duck-typing can silently under-redact non-SecretRef objects**

`isSecretRefShape` only requires two fields (`source: string`, `id: string`) to be present. Any object at an explicitly `sensitive: true` schema path that happens to carry those two string fields will be treated as a SecretRef, and only `id` will be collected/replaced. Any additional sensitive fields on that object (e.g. a co-located `apiKey`, `password`, or `token` field) will be silently skipped by both `redactSecretRef` and the early-return branch in `collectSensitiveStrings`.

The fallback safety check (`shouldFallbackToStructuredRawRedaction`) does not help here: when the structured-redaction path is taken via `redactSecretRef`, the extra fields are already dropped from `values` before the text replacement, so they survive into the `redactedParsed`/`redactedConfig` objects that the fallback stringifies.

A narrower guard — for example also requiring that the object has *no* string fields other than `source`, `provider`, and `id` — would limit this to genuinely SecretRef-shaped values:

```ts
function isSecretRefShape(
  value: Record<string, unknown>,
): value is Record<string, unknown> & { source: string; id: string } {
  const NON_SENSITIVE_SECRETREF_KEYS = new Set(["source", "provider", "id"]);
  const keys = Object.keys(value);
  return (
    typeof value.source === "string" &&
    typeof value.id === "string" &&
    keys.every((k) => NON_SENSITIVE_SECRETREF_KEYS.has(k))
  );
}
```

This ensures the heuristic fires only when the object is structurally exactly a SecretRef, not any wider object that happens to contain `source` and `id`.

How can I resolve this? If you propose a fix, please make it concise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: OpenClaw Config Raw redacts way too much and can't save file

3 participants