Skip to content

HeuristicScorer only checks frontmatter description for triggers — ignores SKILL.md body #223

Description

@diberry

Bug

HeuristicScorer.Score() in internal/scoring/scoring.go checks for trigger patterns (USE FOR:, DO NOT USE FOR:, **UTILITY SKILL**, etc.) only in sk.Frontmatter.Description. It never inspects the SKILL.md body text. Skills that place these routing sections in the body — which is the natural authoring pattern — are scored as Low compliance with "missing triggers" even though the sections are present.

Expected behavior

waza check should detect USE FOR:, DO NOT USE FOR:, and routing clarity markers (**UTILITY SKILL**, INVOKES:, etc.) wherever they appear in SKILL.md — both the frontmatter description and the body.

Actual behavior

waza check reports Low compliance and recommends adding sections that already exist:

Compliance Score: Low
  Needs significant improvement. Description too short or missing triggers.

Next Steps:
  1. Add a 'USE FOR:' section with 3-5 trigger phrases
  2. Add a 'DO NOT USE FOR:' section
  3. Add routing clarity (e.g., **UTILITY SKILL**, INVOKES:)

Root cause

In internal/scoring/scoring.go, lines ~130-140:

desc := sk.Frontmatter.Description
// ...
trimmedDesc := strings.TrimSpace(desc)

result.HasTriggers = containsAny(trimmedDesc, triggerPatterns)
result.HasAntiTriggers = containsAny(trimmedDesc, antiTriggerPatterns)
result.HasRoutingClarity = containsAny(trimmedDesc, routingClarityPatterns)

All three checks use trimmedDesc which is only the frontmatter description field. The body text (sk.Body or equivalent) is never searched.

The computeLevel() function then gates on these booleans:

func computeLevel(r *ScoreResult) AdherenceLevel {
    if r.DescriptionLen < 150 || !r.HasTriggers {
        return AdherenceLow   // <-- always hits this
    }
    // ...
}

Minimal reproduction

Create a SKILL.md with triggers in the body (not the description):

---
name: test-skill
description: A test skill that demonstrates the scoring bug where triggers in the body are not detected by the heuristic scorer compliance check.
allowed-tools: []
---

UTILITY SKILL — Test skill for reproduction.

USE FOR: testing waza compliance scoring, reproducing trigger detection bugs.

DO NOT USE FOR: production use, non-test scenarios.

## References

- [references/example.md](references/example.md) — Example reference

Then run:

waza check test-skill/

Result: Compliance Score: Low, recommends adding USE FOR / DO NOT USE FOR / routing clarity — all of which are present in the body.

Workaround: Stuff all triggers into the frontmatter description string (as skills/waza/SKILL.md does), but this hurts readability and conflicts with the natural body-section authoring pattern.

Suggested fix

In HeuristicScorer.Score(), search both the description and the body:

searchText := trimmedDesc + "\n" + strings.TrimSpace(sk.Body)

result.HasTriggers = containsAny(searchText, triggerPatterns)
result.HasAntiTriggers = containsAny(searchText, antiTriggerPatterns)
result.HasRoutingClarity = containsAny(searchText, routingClarityPatterns)

Or alternatively, parse the body separately and merge the results so trigger counts remain accurate.

Environment

  • waza version: latest (installed via azd extension install waza)
  • OS: Windows 11
  • Discovered while running waza check on 6 Azure SDK review skills that all have USE FOR / DO NOT USE FOR / UTILITY SKILL in the body

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions