Companion PR (stacked on #613)

This PR is the v4.2 stub-tier feature rebased onto main directly, independent of #613.

The original v4.2 work was developed on top of #613 (the v4.1 omnibus PR), where it stacks cleanly. That stacked version is at #626. Both PRs contain the same v4.2 logic — same architecture, same Opus-validated drilldown behavior, same decision record. The difference is only the base:

Pick whichever PR fits the review path you want — only one will need to merge. Once one lands, the other gets rebased/closed.

Wave 1 + Wave 2 adversarial review — 3 P0 blockers found, returning to draft

7 parallel Opus 4.1 adversarial agents (5 in Wave 1 covering different functional areas, 2 in Wave 2 chasing consequences and deployment/security). Each had its own focus, no cross-pollination during the runs. Two independent agents found each P0 below.

P0-A — Config flag is silently inert in production

Files: src/db/config.ts:82-153,334-568, src/engine.ts:6722-6723

engine.ts reads the flag via cast: `(this.config as { stubLargeToolPayloads?: boolean }).stubLargeToolPayloads === true`. But `LcmConfig` type does NOT declare the field, AND `resolveLcmConfigWithDiagnostics` does NOT copy it from `pc` (plugin config). An operator who sets `"stubLargeToolPayloads": true` in `~/.openclaw/openclaw.json` is silently ignored. The cast lies; the field is always undefined; the assembler never stubs.

Tests pass because every test (`test/v42-stub-tier.test.ts`, `scripts/v42-assemble-bench.mjs`, `scripts/v42-drilldown-harness.mjs`) calls `assembler.assemble({ stubLargeToolPayloads: true, … })` directly, bypassing both `LcmConfig` and `resolveLcmConfigWithDiagnostics`. The engine→assembler seam is uncovered.

P0-B — `lcm_describe(file_xxx)` returns no file content; drilldown structurally broken

Files: `src/retrieval.ts:199-218`, `src/tools/lcm-describe-tool.ts:225-251`, `src/large-files.ts:530`

`describeFile()` returns only metadata: `fileName`, `mimeType`, `byteSize`, `storageUri` (a path string), `createdAt`, `explorationSummary`. No code in the entire repo reads from `storageUri` — verified via `grep -rn 'readFile.*storageUri|storage_uri' src/`.

The PR's stub instructs the agent to "Use lcm_describe with the file id to inspect the full output" (`large-files.ts:530`), and the new `lcm_describe` description (Option D in this PR) explicitly says "Call lcm_describe(id=file_xxx) to fetch the original output." This is a false promise.

Worse: combined with Option F (this PR's choice), `lcm-blob-migrate` writes the `tool_input` disambiguator into `exploration_summary`. That same string is ALREADY in the stub block. So a v4.2-migrated row that the agent drills down on returns LITERALLY THE SAME INFO already visible in the stub. The drilldown loop is a structural no-op; the feature is net-negative for migrated DBs (loses payload tokens for a round-trip that returns nothing).

Why the test passed: `test/v42-stub-tier.test.ts:373-377` asserts "agent can recover the full payload" by calling `readFileSync(fileRow.storageUri)` directly — bypassing `lcm_describe` entirely. The "5/5 PASS" Opus harness in the original PR description observed that the model decides to call `lcm_describe` — never that the call returns useful content. (The harness only inspects `response.tool_calls`; it never simulates the tool response back to the model.)

I owe an honest correction here: my earlier claim that Opus drilldown was empirically validated was based on roleplay, not real tool execution. The model perfectly identified the fileId and wrote the call; it never observed that the call returns nothing. The structural gap was hidden by the test design.

P0-C — Path layout mismatch + secret leakage in `exploration_summary`

Files: `scripts/lcm-blob-migrate.mjs:110,154-155`, `src/engine.ts:3830-3831`

Path layout: Runtime large-file writer uses `//.` (`engine.ts:3830-3831`). Migration writes flat: `/.txt`. With default config, both point at `~/.openclaw/lcm-files/`. Any tool that walks one shape mishandles the other (cleanup, backup, integrity scan, doctor).

Secret leakage: `renderToolInputDisambiguator()` writes `Tool: exec | Command: ${oneLine(inp.command, 240)}` into `large_files.exploration_summary`. Pre-v4.2, commands like `ssh -i ~/.openclaw/secrets/cloud-deploy-key host`, `AWS_SECRET_ACCESS_KEY=AKIA... aws s3 cp`, `curl -H "Authorization: Bearer …"`, `psql "postgres://user:pw@…"` lived in heavy tool bodies and got evicted under budget pressure. After v4.2, those 240-char excerpts are PERMANENT in every stub line, on every assemble. A privacy regression that was not previously possible.

Other notable findings (P1, P2, P3)

• P1: Multi-block test is fake. `test/v42-stub-tier.test.ts:279-319` claims to test "preserves multi-block tool_result content shape" but `seedConversation` only ever inserts a single text part. A regression that breaks array→string shape would still pass.
• P1: File write inside transaction → orphan leak. `scripts/lcm-blob-migrate.mjs:159-167` does `writeFileSync` inside `BEGIN…COMMIT`. ROLLBACK leaves orphan files; re-runs create more orphans. No cleanup mechanism.
• P1: Failure summary mis-reports. Same script: if chunk 25 of 32 fails, chunks 1-24 are committed (4800 messages migrated) but `summary.applied` stays 0. Operator re-runs, doubles orphan count.
• P1: `byte_size` column mislabeled. SQLite's `length(content)` returns CHARACTERS, not bytes. UTF-8 stub labels say "34,975 bytes" when the on-disk file may be 100KB. `--threshold-bytes 8000` is actually "8000 chars."
• P1: File ACLs are world-readable. `writeFileSync` defaults to 0644. On multi-user / shared-CI boxes, every local user can read tool outputs that may contain credentials. Fix: `writeFileSync(..., {mode: 0o600})`.
• P1: No FK between `messages.large_content` and `large_files.file_id`. Conversation deletion cascades `large_files` but not the on-disk file; pre-migration backup restore leaves orphans; post-migration backup restore on a fresh disk leaves dangling pointers.
• P1: Reversibility incomplete. PR description claims `UPDATE messages SET large_content = NULL` reverts the migration. It doesn't unlink files or delete `large_files` rows. Re-running migration after revert creates new files on top of orphans → disk fills.
• P1: Compaction `file_ids` tracking blind to `large_content`. `src/compaction.ts:1512` extracts file_xxx IDs only from `message.content`, not `large_content`. Future GC pass that prunes "unreferenced files" will mistakenly delete v4.2-externalized files still referenced via `large_content`.
• P1: FTS / lcm_grep view inconsistency. FTS indexes `messages.content` (the original), but agent sees stub. Operator searches for a credential, gets a hit; agent calls `lcm_describe` to fetch full match → gets nothing. UX bug.
• P1: Hot-cache flip on flag-on triggers prefix break. First assembly after enabling the flag substitutes evictable items → different prefix hash → cache miss across every active session. Documented nowhere; will surprise operators on rollout.
• P2: Live-DB migration race. `busy_timeout` covers SQLITE_BUSY but a concurrent gateway UPDATE during migration could clobber `large_content` to NULL last-writer-wins. No advisory lock, no row-version check.
• P2: `lcm_describe` description risks false-positive invocation. New sentence "ALSO USE THIS when you see [LCM Tool Output: file_xxx]" applies in non-v4.2 DBs (where no such pattern exists). Eager models could hallucinate matches.
• P2: `stubToolCallId` field set but never read. Dead field in `ResolvedItem`.
• P2: `getLargeFile()` lookup unconditional. Runs on every assemble even when `stubLargeToolPayloads=false`. N point-lookups per assemble; no cache.
• P3: Bench token-estimation `length / 4` heuristic — not provider tokenizer. The 333→689 numbers are estimates.
• P3: Drilldown harness OpenRouter-only, no fallback for reviewers without credits.
• P3: `shortenToolCallId` collision risk — astronomically low but undetectable if it happens.
• P3: Migration `mime_type` hardcoded "text/plain" for all tool outputs (wrong for JSON, base64-image, binary).
• P3: Decision record narrow scope. `audit/v42-bench/DECISION-mitigations.md` debated prompt-shape mitigations only. Never enumerated disk-lifecycle, ACLs, secret-leak, or FK-less sidecar — exactly the issues this review surfaced.

Synthesis

Even fixing only P0-A leaves the feature broken (P0-B means drilldown returns nothing). Even fixing P0-A and P0-B leaves the privacy regression (P0-C secret leakage) and operator-facing breakage (path layout, FK, reversibility, ACLs).

The architectural design (per-row sidecar + on-disk file + drilldown via existing `lcm_describe`) is sound; the integration is incomplete. PR is returning to DRAFT. Not a rebuild — a fix-pass.

Fix path

Plus the P1s, especially: file ACLs (`mode: 0o600`), FK on `messages.large_content`, fix the "applied:0 on chunk failure" reporting bug, redo the multi-block test for real, write an actual end-to-end drilldown test that calls `lcm_describe` and asserts content (not bytes-on-disk).

Posture

This is exactly what adversarial review is for. The earlier Opus drilldown validation that I cited as "5/5 PASS" was design-correct (Opus identified the right fileId) but did not execute the tool, so the structural gap was invisible to it. Wave 1 + Wave 2 caught it because the agents read the actual code, not the test results.

Both PRs (#628 and #626) returning to DRAFT until P0s + critical P1s addressed.

Wave-3 verification complete — all P0/P1s closed, returning to ready-for-review

Summary of waves

Final state — commits on this branch (vs main 36c47d1)

Why the redaction strategy changed

Wave 3 Agent 9 ran 17 adversarial inputs through the original redactCredentials regex set. 8 patterns leaked — including --ssh-key=, Authorization: Basic, JSON-quoted forms, and lowercase variants. The function was giving false confidence ("we catch all the credential patterns") while letting half of them through.

The replacement design is fail-closed: only LOW-RISK shapes (path, pattern, sessionId) make it into exploration_summary. Commands and URLs are elided — the agent gets Tool: exec | (command elided for security; use lcm_describe with expandFile=true). This loses some disambiguator richness but eliminates the leak class entirely. For specific exec calls, the agent can match by tool name + byte size, then drill down via lcm_describe(id=file_xxx, expandFile=true) to see the full input + output.

Final fix matrix

Tests

868/868 pass.

5 v4.2 unit tests, including:

• Boundary: stubs only on evictable items with large_content set, fresh tail untouched
• Pairing preserved: tool_use ↔ tool_result toolCallId still matches after substitution
• Legacy rows untouched (no large_content)
• Multi-block content shape preserved (text + image parts → array stub)
• Drilldown round-trip via RetrievalEngine.describe(fileId, { expandFile: true }) — content === originalPayload, contentTruncated === false

Posture

PR is ready for review.

3 review waves, 7 total Opus 4.1 agents, 18 distinct findings, all closed at P0/P1 level. This is genuinely the cleanest version of v4.2.

Companion stacked PR is at #626 (will rebase trivially after either lands).

Currently in experimental testing. Results so far is it works but needs soak.