Pull request overview

This PR adds a new CLI workflow to generate task.yaml files by recording a Copilot prompt run (waza new task record), and refactors waza new into subcommands (notably waza new skill). It also updates docs/examples to reflect the new CLI shape and improves grader-type error reporting.

Changes:

• Introduce waza new task record to execute a prompt via Copilot, read the session log, and emit a generated task YAML with inferred validators.
• Restructure waza new to use waza new skill (and update docs/tests accordingly).
• Add a helper to list valid grader kinds and improve unsupported-grader error messages.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 15 comments.

		case copilot.ToolExecutionComplete:
			t := tools[*e.Data.ToolCallID]
			t.End = e.Timestamp
			t.Success = *e.Data.Success

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 10 comments.

		case copilot.ToolExecutionStart:
			toolsInOrder = append(toolsInOrder, *e.Data.ToolCallID)

			if name == nil {
				name = new("<no skill name>")
			}

		case copilot.ToolExecutionComplete:
			t := tools[*e.Data.ToolCallID]
			t.End = e.Timestamp
			t.Success = *e.Data.Success

			tools[*e.Data.ToolCallID] = &tool{
				Start:     e.Timestamp,
				Name:      *e.Data.ToolName,
				Arguments: *ta,
			}

waza new skill code-analyzer << EOF
Code Analyzer
Analyzes code for patterns and issues
code, analysis
EOF

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 4 comments.

You can also share your feedback on Copilot code review. Take the survey.

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 1 comment.

			path := e.Data.Path
			if path == nil {
				path = new("<no path>")
			}

You can also share your feedback on Copilot code review. Take the survey.

Code Review: PR #110 — Adding in wz new task record

✅ What Looks Good

• Clean interface export — copilotClient → CopilotClient properly exports interfaces for external mock generation
• Good use of Go iterators — NewCopilotLogIterator uses iter.Seq2 idiomatically with proper error handling
• Improved grader error messages — AllGraderKinds() provides helpful valid-types list on invalid grader config
• Comprehensive mock infrastructure — go:generate setup with proper DI via options struct
• Solid E2E test — cmd_new_task_test.go covers the full happy path with mocked dependencies

🟠 High (4 findings)

1. Consider from-prompt as default behavior or flag instead of subcommand — Currently waza new task from-prompt <prompt> <path> uses from-prompt as a subcommand. If there's only one way to create tasks right now, consider making it the default behavior of waza new task (with --from-prompt as a flag if more sources are added later). Running waza new task with no subcommand currently shows help — would a default action be more ergonomic?

2. Brittle default text grader — entire response as ContainsCS — converters.go:215-223 — The generated task YAML uses the full concatenated assistant response as a case-sensitive contains check. LLM responses are non-deterministic, so this grader will fail on re-runs. Consider extracting key phrases instead, or adding a YAML comment noting this should be customized.

3. converters.go — 1 test for 218 lines of code — CreateTestCaseFromCopilotLog handles 6 event types (UserMessage, ToolExecutionStart/Complete, AssistantMessage/Delta, SkillInvoked) but has only 1 happy-path test. Missing coverage for: nil ToolCallID, mapstructure decode failures, tool-not-found in map, report_intent filter, nil skill name/path defaults.

4. Hardcoded default model claude-sonnet-4.5 — cmd_new_task.go:291 — Model name hardcoded as flag default. Consider reading from config/env (WAZA_DEFAULT_MODEL) or deferring to the Copilot SDK's own default.

🟡 Medium (4 findings)

5. --testname should be --test-name — All other multi-word flags use kebab-case (--disable-repo-skills, --judge-model). This one doesn't.

6. context.Background() in shutdown defer — cmd_new_task.go:165 — Creates a new background context instead of inheriting cmd.Context(). If user presses Ctrl+C, shutdown still waits up to 1 minute.

7. RunE closure is ~160 lines — cmd_new_task.go:129-288 — Single function handles 6 different concerns. Consider extracting task definitions into helper functions for readability and testability.

8. new("<no skill name>") shadows Go builtin — converters.go:158,163 — Confusing for readers who expect new(T) to allocate a zero-value pointer. Use explicit s := "<no skill name>"; name = &s instead.

🟢 Low (2 findings)

9. Missing error context in skill discovery — cmd_new_task.go:182-186 — Wrap error with: fmt.Errorf("failed to discover skills in %q: %%w", rootDir, err)

10. Task list wrapper swallows AddTask error — cmd_new_task.go:324 — _ = tlw.inner.AddTask(options) silently ignores errors.

📌 Documentation Pass Recommended

After this PR lands, recommend a focused round of doc updates to ensure all docs match the new waza new command structure (waza new skill, waza new task, waza new eval). Some references to the old waza new [skill-name] and waza generate patterns may still exist across README, guides, and site content.

Summary

Overall Assessment: Comment — solid new feature with good DI/mock patterns. Key items to address: the brittle text grader default, test coverage for converters, and flag naming consistency.

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

You can also share your feedback on Copilot code review. Take the survey.

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 5 comments.

You can also share your feedback on Copilot code review. Take the survey.