Pull request overview

Adds a new waza eval new <skill-name> CLI subcommand to scaffold an evaluation suite from a skill’s SKILL.md frontmatter, plus docs/tests to make it discoverable and verifiable.

Changes:

• Introduce waza eval new command that parses SKILL.md triggers and generates eval.yaml + starter trigger/anti-trigger task YAMLs.
• Add unit tests validating scaffold generation, custom output path behavior, and missing SKILL.md error handling.
• Update CLI docs + README + command metadata expectations to include the new eval top-level command.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

	if outputPath == "" {
		outputPath = filepath.Join("evals", skillName, "eval.yaml")
	}
	tasksDir := filepath.Join(filepath.Dir(outputPath), "tasks")

| Flag | Description |
|------|-------------|
| `--output` | Custom path for generated `eval.yaml` |

LGTM — Rusty. waza eval new is a clean scaffolding command. Good use of SKILL.md frontmatter parsing for positive/negative trigger generation. extractKeywords with stop words is smart. Tests validate generated YAML through validation.ValidateEvalBytes/ValidateTaskBytes — nice. README + CLI reference updated. Ship it. (Self-authored PR — cannot self-approve via API.)

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Code Review: PR #94 - feat: Add eval scaffolding command (waza eval new)

What Looks Good

• Smart SKILL.md resolution: workspace-aware detection with .waza.yaml config, fallback to conventional paths
• Safe overwrite protection and partial write cleanup
• Generated YAML validated in tests via ValidateEvalBytes/ValidateTaskBytes
• Custom output path works correctly
• Docs fully updated

Discussion: Command Overlap

Before approving, I want to flag a naming/scope discussion:

1. waza new already creates a skill + eval scaffold together
2. waza eval new (this PR) creates eval scaffold from an existing skill
3. waza init detects skills missing evals and can create them following config

Questions worth resolving:

• Should this be waza new eval (verb-noun pattern) instead of waza eval new? The verb-noun pattern (waza new [skill|eval]) is more discoverable and consistent.
• Does waza init's gap-filling already cover the use case of adding evals to existing skills? If so, is a separate command needed or would init --skill=name suffice?
• If we keep eval as a subcommand group, what other subcommands go under it? If it's just new, the group may be premature.

Findings

Medium:

1. Hardcoded 2 positive + 1 negative task structure may not suit skills with rich USE FOR sections (5+ phrases) or minimal descriptions.

Low:
2. Site docs use [skill-name] (optional) instead of (required).
3. extractKeywords stop words overlap with trigger_grader.go stop words.

Summary

Overall Assessment: Comment - implementation is solid, but the command naming and overlap with waza new and waza init should be discussed before merging.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

	if outputPath == "" {
		outputPath = filepath.Join("evals", skillName, "eval.yaml")
	}

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.