feat: add skill_brain_first doctor check#1206
Conversation
Scans SKILL.md files in the OpenClaw workspace to detect skills that reference external lookup tools (web_search, Exa, Perplexity, Happenstance, Crustdata, Captain API, etc.) without also including a brain/gbrain search step. The brain has 100K+ pages — external APIs should never be the first call when the brain likely has the answer. Motivated by the tweet-shield incident (2026-05-19): cross-modal eval flagged Garry's Palantir tweet as risky because models didn't know he built it, but the brain already had 'designed the entire Finance product UI' and '150+ PSDs from April-December 2006.' The check: - Scans all SKILL.md files in the OpenClaw workspace - Detects external lookup patterns (web_search, exa, perplexity, etc.) - Verifies those skills also include a brain-first search step - Reports [WARN] with list of non-compliant skills - Exempts brain-internal and pure-infra skills Includes 10 unit tests covering all edge cases.
|
Closing — superseded by #1215 (v0.37.1.0 skill_brain_first cathedral wave). The brain-first miss class you surfaced (the tweet-shield framing in your PR Structural changes from your version:
Your 10 unit cases are absorbed verbatim into the v0.37.1.0 fixture corpus
|
…ve opt-out (supersedes #1206) (#1215) * v0.37.1.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out Cathedral wave superseding PR #1206. Doctor now scans every SKILL.md for external-lookup tools (web_search / web_fetch / exa / perplexity / happenstance / crustdata / captain_api / firecrawl) and warns when the skill has no brain- first compliance signal. gbrain doctor --fix auto-inserts the canonical > **Convention:** see [conventions/brain-first.md](...) callout via the dry-fix.ts MISSING_RULE_PATTERNS extension (sharing safety gates with the existing REPLACE patterns). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches authorship; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. Key design decisions locked via /plan-eng-review + codex outside-voice review: - A1: frontmatter ships only brain_first: exempt (no required/n/a enum) - A2: snapshot+diff audit at ~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl with transition-only writes (stable brains = 0 lines/run) - A3: scaffold template pre-inserts callout; skillify check fails (exit 1) on external + no callout + no exempt - A4: position-relative gate is BODY-ONLY (frontmatter tools: [web_search] declaration doesn't false-flag the skill) - Q1: single pure analyzeSkillBrainFirst() helper consumed by 3 surfaces - CMT1: no upgrade migration — doctor surfaces hint, --fix applies via dry-fix safety gates (user stays in loop) - CMT2: dropped tools+writes_pages auto-exemption (was hiding mixed-class skills like idea-ingest/meeting-ingestion/data-research) Trio: VERSION + package.json + CHANGELOG aligned at 0.37.1.0. 56 unit cases + 12 E2E cases pass. 170 related existing tests pass unchanged. Self-dogfood: gbrain doctor against this repo's skills/ reports skill_brain_first: ok across 43 skills (compliant or exempt). functional-area-resolver and strategic-reading skills gained brain_first: exempt to validate the declarative opt-out in production code (both name perplexity in dispatcher prose without calling it). Co-Authored-By: garrytan-agents <noreply@github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md for v0.37.1.0 skill_brain_first wave Added Key Files entries for the four new modules: - src/core/skill-frontmatter.ts (shared parser) - src/core/skill-brain-first.ts (analyzer + FORMERLY_HARDCODED_EXEMPT) - src/core/skill-fix-gates.ts (extracted safety primitives) - src/core/audit-skill-brain-first.ts (snapshot+diff JSONL) Extended existing entries: - src/core/filing-audit.ts: rewired to shared parser - src/core/dry-fix.ts: MISSING_RULE_PATTERNS INSERT pattern type - src/commands/doctor.ts: skill_brain_first check + tweet-shield framing - src/commands/skillify-check.ts: required item 12 + scaffold pre-insert Added test inventory entries: - test/skill-brain-first.test.ts (56 unit cases) - test/e2e/skill-brain-first.test.ts (12 E2E cases) Regenerated llms-full.txt via bun run build:llms. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): skill_brain_first guard uses doctor --fast to skip engine connect CI run #76881161092 failed because scripts/check-skill-brain-first.sh invoked plain `gbrain doctor --json`, which routes through connectEngine(). With no ~/.gbrain/config.json present (CI's case — runner is bun-only, no brain init), connectEngine() exits 1 with "No brain configured." and emits zero stdout. The python parser sees an empty file and returns parse_error, failing the verify gate. Fix: pass --fast to doctor. --fast routes through runDoctor(null, ...) which runs the filesystem-only check set (resolver_health, skill_conformance, skill_brain_first) and emits the standard single-line JSON envelope the parser expects. skill_brain_first is filesystem-only by design (scans SKILL.md, no DB touch), so --fast is the correct knob, not a workaround. Verified by reproducing the CI failure mode locally with GBRAIN_HOME=/tmp/empty-... — gate now passes both with and without a configured brain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.37.1.0 → v0.37.3.0 (queue collision with #1214) PR #1214 (brainstorm + lsd) claimed v0.37.1.0 concurrently with #1215. Skipping 0.37.2.0 leaves a buffer for #1214's adjacent slot. Trio (VERSION + package.json + CHANGELOG header + inline "To take advantage of v0.37.3.0" block) aligned at 0.37.3.0. No behavior changes — version metadata only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <noreply@github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* upstream/master: v0.38.2.0 fix(doctor): bounded frontmatter scan + partial-state surfacing (supersedes garrytan#1287) (garrytan#1297) v0.38.1.0 feat(agents): provider-agnostic subagent loop + remote MCP dispatch + budget meter (garrytan#1289) v0.38.0.0 ingestion cathedral — gbrain capture + write-through + IngestionSource contract (garrytan#1275) v0.37.11.0: fresh-install PGLite embedding setup fix wave (garrytan#1286) v0.37.10.0 feat(init): env-detection + interactive picker + preflight invariants (garrytan#1278) v0.37.9.0 fix(frontmatter): canonical-style normalization for tag arrays (garrytan#1252) v0.37.8.0 feat: voyage-code-3 discoverability + reindex-code cost-preview fix (garrytan#1267) v0.37.7.0 fix wave: federated brains + autopilot safety + OAuth confidential clients (garrytan#1253) v0.37.6.0 feat(ai): OpenRouter recipe + generic default_headers seam (cherry-pick garrytan#1210) (garrytan#1246) v0.37.5.0 fix(markdown): YAML-aware NESTED_QUOTES validator (stops flagging valid YAML) (garrytan#1229) feat: pgGraph-inspired CI scaffolding wave (v0.37.4.0) (garrytan#1228) v0.37.3.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out (supersedes garrytan#1206) (garrytan#1215) v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (garrytan#1211) v0.37.1.0 feat: brainstorm + lsd — bisociation idea generator grounded in your own brain (garrytan#1214) v0.37.0.0 feat(skillpack): registry cathedral — third-party publish + install + 10/10 quality bar (garrytan#1208) v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (garrytan#1165)
What
Adds a new
skill_brain_firstcheck togbrain doctorthat scans SKILL.md files in the workspace and detects skills doing external lookups without a brain-first search step.Why
A content evaluation skill ran cross-modal analysis on user-authored content without first checking the brain for relevant context. All 3 evaluation models flagged the content as risky because they relied on incomplete public knowledge — but the brain already had first-person records confirming the claim. If the skill had searched the brain first, it would never have false-flagged.
The brain can have 100K+ pages of verified context. External APIs should never be the first call when the brain likely has the answer. This doctor check enforces that pattern systematically.
How
The check runs in the filesystem-only section of
runDoctor(no DB needed):SKILL.mdfiles in the workspace ($OPENCLAW_WORKSPACE/skillsor~/.openclaw/workspace/skills)web_search,web_fetch, Exa, Perplexity, Happenstance, Crustdata, Captain API, Firecrawlgbrain search,brain search,brain-first,Step 0: Brain Context, etc.)[WARN]with a sorted list of non-compliant skillsExample output
Tests
10 unit tests in
test/doctor-brain-first.test.ts:All passing.