Software-grade release discipline for Claude Skills

Binary evaluation harness that treats SKILL.md artifacts as production software. Package integrity, trigger precision, functional quality, regression gating, baseline comparison, model-aware testing, and evidence-backed rollout decisions — all through binary yes/no criteria with external evaluators.

Links: Master Blueprint · Epic Index · Doc Index · Contributing · Security

Claude Skills ship on instinct. A skill author writes a SKILL.md, eyeballs it, maybe runs it once, and pushes. There is no regression gate, no trigger precision measurement, no baseline comparison, no model-variance tracking, and no evidence trail for rollout decisions. When a skill breaks silently after a model update or a description tweak causes sibling confusion across a pack, nobody knows until users complain.

J-Rig Binary Eval is a seven-layer evaluation harness that scores every skill change across seven product surfaces before it ships:

1. Package Integrity — Does it parse, validate, and reference real files?
2. Trigger Quality — Does it fire on the right prompts and stay silent on the wrong ones?
3. Functional Quality — Does it complete its task and produce correct artifacts?
4. Regression Protection — Did this change break anything that previously worked?
5. Baseline Value — Does the skill actually outperform the naked model?
6. Model Variance — Does it work across Haiku, Sonnet, and Opus?
7. Rollout Safety — Any prompt leakage, overreach, or unsafe automation?

Every criterion is binary (yes/no). The evaluator is always separate from the skill under test. Observed behavior outranks claimed behavior.

• Binary criteria only — if a criterion can't be answered yes or no, it isn't ready. No fuzzy scores, no vibes.
• External evaluators — the skill under test never judges itself. Deterministic checks first, LLM judges second.
• Sacred regressions — a change that improves average score but breaks a sacred case is rejected. Period.
• One change at a time — the optimizer proposes exactly one atomic change per experiment. No multi-variable confusion.
• Baseline gating — if the base model already does the job without the skill, the skill gets flagged for obsolete review.
• Model-aware — Haiku, Sonnet, and Opus are tested independently. Model variance is product reality, not noise.
• Evidence-backed rollout — every ship/no-ship decision comes with a structured evidence trail.

┌─────────────────────────────────────────────────┐
│                   CLI / CI / API                 │  Layer 7: Surfaces
├─────────────────────────────────────────────────┤
│                 Evidence Layer                   │  Layer 6: Persistence
├─────────────────────────────────────────────────┤
│               Optimization Layer                 │  Layer 5: Experiments
├─────────────────────────────────────────────────┤
│                Judgment Layer                    │  Layer 4: Scoring
├─────────────────────────────────────────────────┤
│              Observation Layer                   │  Layer 3: Capture
├─────────────────────────────────────────────────┤
│               Execution Layer                    │  Layer 2: Harness
├─────────────────────────────────────────────────┤
│                  Spec Layer                      │  Layer 1: Contracts
└─────────────────────────────────────────────────┘

1. Criteria must be binary — yes or no, no gradients
2. Evaluator is always separate — the skill never judges itself
3. Observed behavior outranks claimed behavior — grade what happened, not what the skill says it does
4. Regression tests are sacred — a regression on a sacred case blocks release regardless of average improvement
5. One change at a time — optimizer proposes exactly one atomic change per experiment
6. Blockers block release — a blocker failure cannot be averaged out
7. Baseline value matters — if the naked model matches the skill, flag for obsolete review
8. Model-aware testing is required — Haiku/Sonnet/Opus differences are product reality

Self-contained library of templates, reference standards, agent patterns, and workflow diagrams under 000-docs/:

Phase: Epic 01 complete (repo foundation). Ready for Epic 02 (Spec Layer).

pnpm monorepo with four workspace packages (@j-rig/core, @j-rig/cli, @j-rig/db, @j-rig/dashboard), TypeScript baseline (tsup builds), quality guardrails (ESLint, Prettier, Vitest), and CI/CD workflows.

MIT — see LICENSE for details.

Jeremy Longshore — jeremylongshore · Intent Solutions