Skip to content

feat: add skill_brain_first doctor check#1206

Closed
garrytan-agents wants to merge 1 commit into
garrytan:masterfrom
garrytan-agents:feature/doctor-brain-first-check
Closed

feat: add skill_brain_first doctor check#1206
garrytan-agents wants to merge 1 commit into
garrytan:masterfrom
garrytan-agents:feature/doctor-brain-first-check

Conversation

@garrytan-agents

@garrytan-agents garrytan-agents commented May 19, 2026

Copy link
Copy Markdown
Contributor

What

Adds a new skill_brain_first check to gbrain doctor that scans SKILL.md files in the workspace and detects skills doing external lookups without a brain-first search step.

Why

A content evaluation skill ran cross-modal analysis on user-authored content without first checking the brain for relevant context. All 3 evaluation models flagged the content as risky because they relied on incomplete public knowledge — but the brain already had first-person records confirming the claim. If the skill had searched the brain first, it would never have false-flagged.

The brain can have 100K+ pages of verified context. External APIs should never be the first call when the brain likely has the answer. This doctor check enforces that pattern systematically.

How

The check runs in the filesystem-only section of runDoctor (no DB needed):

  1. Scans all SKILL.md files in the workspace ($OPENCLAW_WORKSPACE/skills or ~/.openclaw/workspace/skills)
  2. Detects external lookup patterns: web_search, web_fetch, Exa, Perplexity, Happenstance, Crustdata, Captain API, Firecrawl
  3. Checks if those skills also include a brain-first search step (gbrain search, brain search, brain-first, Step 0: Brain Context, etc.)
  4. Reports [WARN] with a sorted list of non-compliant skills
  5. Exempts brain-internal skills (brain-ops, gbrain, etc.) and pure-infra skills

Example output

[WARN] skill_brain_first: 42 skill(s) do external lookups without a brain-first search step

Tests

10 unit tests in test/doctor-brain-first.test.ts:

  • Directory not found → ok (skipped)
  • All external-lookup skills have brain-first → ok
  • Missing brain-first → warn with skill names
  • Multiple external patterns detected
  • Various brain-first pattern recognition
  • Exempt skills skipped
  • Hidden/underscore dirs skipped
  • Mix of good/bad reports only bad

All passing.

Scans SKILL.md files in the OpenClaw workspace to detect skills that
reference external lookup tools (web_search, Exa, Perplexity,
Happenstance, Crustdata, Captain API, etc.) without also including a
brain/gbrain search step.

The brain has 100K+ pages — external APIs should never be the first
call when the brain likely has the answer.

Motivated by the tweet-shield incident (2026-05-19): cross-modal eval
flagged Garry's Palantir tweet as risky because models didn't know he
built it, but the brain already had 'designed the entire Finance product
UI' and '150+ PSDs from April-December 2006.'

The check:
- Scans all SKILL.md files in the OpenClaw workspace
- Detects external lookup patterns (web_search, exa, perplexity, etc.)
- Verifies those skills also include a brain-first search step
- Reports [WARN] with list of non-compliant skills
- Exempts brain-internal and pure-infra skills

Includes 10 unit tests covering all edge cases.
@garrytan

Copy link
Copy Markdown
Owner

Closing — superseded by #1215 (v0.37.1.0 skill_brain_first cathedral wave).

The brain-first miss class you surfaced (the tweet-shield framing in your PR
body) carries forward verbatim into v0.37.1.0's CHANGELOG and motivates the
entire wave. Your EXEMPT_SKILLS list survives as the
FORMERLY_HARDCODED_EXEMPT constant in src/core/skill-brain-first.ts
informational only (drives the upgrade-time doctor hint), not an exemption
rule, so it doesn't rot.

Structural changes from your version:

  • Hardcoded 40-name allowlist → structural-signal inference + declarative
    brain_first: exempt frontmatter opt-out
  • No --fix path → dry-fix.ts MISSING_RULE_PATTERNS extension that
    auto-inserts the canonical > **Convention:** see [conventions/brain-first.md](...)
    callout (with full safety gates: working-tree-clean, code-fence, install-path)
  • Flat warning → structured Check.issues[] array + snapshot+diff JSONL audit
  • Bypassed resolver → canonical autoDetectSkillsDirReadOnly()
  • No regression prevention → scaffold pre-inserts callout + skillify check
    fails exit-1 on external+no-callout+no-exempt

Your 10 unit cases are absorbed verbatim into the v0.37.1.0 fixture corpus
(IRON-RULE regression preservation). Thank you for the seed — the cathedral
wouldn't have been framed this clearly without your tweet-shield write-up.

Co-Authored-By: garrytan-agents lands on the v0.37.1.0 commit when #1215
merges.

@garrytan garrytan closed this May 20, 2026
garrytan added a commit that referenced this pull request May 20, 2026
…ve opt-out (supersedes #1206) (#1215)

* v0.37.1.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out

Cathedral wave superseding PR #1206. Doctor now scans every SKILL.md for
external-lookup tools (web_search / web_fetch / exa / perplexity / happenstance
/ crustdata / captain_api / firecrawl) and warns when the skill has no brain-
first compliance signal. gbrain doctor --fix auto-inserts the canonical
> **Convention:** see [conventions/brain-first.md](...) callout via the
dry-fix.ts MISSING_RULE_PATTERNS extension (sharing safety gates with the
existing REPLACE patterns).

Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged
Garry's Palantir tweet as risky because no model knew he built it, but the
brain already had "designed the entire Finance product UI" and "150+ PSDs
from April-December 2006." Static check catches authorship; v0.37+ runtime
gate (filed in TODOS.md) closes the dispatch side.

Key design decisions locked via /plan-eng-review + codex outside-voice review:
- A1: frontmatter ships only brain_first: exempt (no required/n/a enum)
- A2: snapshot+diff audit at ~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl
  with transition-only writes (stable brains = 0 lines/run)
- A3: scaffold template pre-inserts callout; skillify check fails (exit 1)
  on external + no callout + no exempt
- A4: position-relative gate is BODY-ONLY (frontmatter tools: [web_search]
  declaration doesn't false-flag the skill)
- Q1: single pure analyzeSkillBrainFirst() helper consumed by 3 surfaces
- CMT1: no upgrade migration — doctor surfaces hint, --fix applies via
  dry-fix safety gates (user stays in loop)
- CMT2: dropped tools+writes_pages auto-exemption (was hiding mixed-class
  skills like idea-ingest/meeting-ingestion/data-research)

Trio: VERSION + package.json + CHANGELOG aligned at 0.37.1.0. 56 unit cases
+ 12 E2E cases pass. 170 related existing tests pass unchanged. Self-dogfood:
gbrain doctor against this repo's skills/ reports skill_brain_first: ok
across 43 skills (compliant or exempt). functional-area-resolver and
strategic-reading skills gained brain_first: exempt to validate the
declarative opt-out in production code (both name perplexity in dispatcher
prose without calling it).

Co-Authored-By: garrytan-agents <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md for v0.37.1.0 skill_brain_first wave

Added Key Files entries for the four new modules:
- src/core/skill-frontmatter.ts (shared parser)
- src/core/skill-brain-first.ts (analyzer + FORMERLY_HARDCODED_EXEMPT)
- src/core/skill-fix-gates.ts (extracted safety primitives)
- src/core/audit-skill-brain-first.ts (snapshot+diff JSONL)

Extended existing entries:
- src/core/filing-audit.ts: rewired to shared parser
- src/core/dry-fix.ts: MISSING_RULE_PATTERNS INSERT pattern type
- src/commands/doctor.ts: skill_brain_first check + tweet-shield framing
- src/commands/skillify-check.ts: required item 12 + scaffold pre-insert

Added test inventory entries:
- test/skill-brain-first.test.ts (56 unit cases)
- test/e2e/skill-brain-first.test.ts (12 E2E cases)

Regenerated llms-full.txt via bun run build:llms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): skill_brain_first guard uses doctor --fast to skip engine connect

CI run #76881161092 failed because scripts/check-skill-brain-first.sh
invoked plain `gbrain doctor --json`, which routes through connectEngine().
With no ~/.gbrain/config.json present (CI's case — runner is bun-only,
no brain init), connectEngine() exits 1 with "No brain configured." and
emits zero stdout. The python parser sees an empty file and returns
parse_error, failing the verify gate.

Fix: pass --fast to doctor. --fast routes through runDoctor(null, ...)
which runs the filesystem-only check set (resolver_health,
skill_conformance, skill_brain_first) and emits the standard
single-line JSON envelope the parser expects. skill_brain_first is
filesystem-only by design (scans SKILL.md, no DB touch), so --fast is
the correct knob, not a workaround.

Verified by reproducing the CI failure mode locally with
GBRAIN_HOME=/tmp/empty-... — gate now passes both with and without
a configured brain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump v0.37.1.0 → v0.37.3.0 (queue collision with #1214)

PR #1214 (brainstorm + lsd) claimed v0.37.1.0 concurrently with #1215.
Skipping 0.37.2.0 leaves a buffer for #1214's adjacent slot. Trio
(VERSION + package.json + CHANGELOG header + inline "To take advantage
of v0.37.3.0" block) aligned at 0.37.3.0.

No behavior changes — version metadata only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <noreply@github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mgunnin added a commit to mgunnin/gbrain that referenced this pull request May 28, 2026
* upstream/master:
  v0.38.2.0 fix(doctor): bounded frontmatter scan + partial-state surfacing (supersedes garrytan#1287) (garrytan#1297)
  v0.38.1.0 feat(agents): provider-agnostic subagent loop + remote MCP dispatch + budget meter (garrytan#1289)
  v0.38.0.0 ingestion cathedral — gbrain capture + write-through + IngestionSource contract (garrytan#1275)
  v0.37.11.0: fresh-install PGLite embedding setup fix wave (garrytan#1286)
  v0.37.10.0 feat(init): env-detection + interactive picker + preflight invariants (garrytan#1278)
  v0.37.9.0 fix(frontmatter): canonical-style normalization for tag arrays (garrytan#1252)
  v0.37.8.0 feat: voyage-code-3 discoverability + reindex-code cost-preview fix (garrytan#1267)
  v0.37.7.0 fix wave: federated brains + autopilot safety + OAuth confidential clients (garrytan#1253)
  v0.37.6.0 feat(ai): OpenRouter recipe + generic default_headers seam (cherry-pick garrytan#1210) (garrytan#1246)
  v0.37.5.0 fix(markdown): YAML-aware NESTED_QUOTES validator (stops flagging valid YAML) (garrytan#1229)
  feat: pgGraph-inspired CI scaffolding wave (v0.37.4.0) (garrytan#1228)
  v0.37.3.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out (supersedes garrytan#1206) (garrytan#1215)
  v0.37.2.0: takes_resolution_consistency CHECK accepts 'unresolvable' (garrytan#1211)
  v0.37.1.0 feat: brainstorm + lsd — bisociation idea generator grounded in your own brain (garrytan#1214)
  v0.37.0.0 feat(skillpack): registry cathedral — third-party publish + install + 10/10 quality bar (garrytan#1208)
  v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (garrytan#1165)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants