Summary
Add a --update-snapshots flag to waza run that updates diff grader baseline files when they've drifted, similar to Jest's --updateSnapshot or Go's -update flag.
Motivation
The diff_grader already compares workspace files against expected snapshots. But when intentional changes are made, manually updating all snapshot files is tedious. An auto-update mode captures the current output as the new baseline.
Proposed Implementation
Flag: waza run eval.yaml --update-snapshots
When enabled:
- Run tasks normally
- For any diff grader that detects differences, instead of failing:
- Copy the actual output to the expected snapshot path
- Mark the task as "updated" (not pass/fail)
- Report which snapshots were updated
Output
📸 Snapshot Updates:
✏️ fixtures/expected/output.md — updated (3 lines changed)
✏️ fixtures/expected/config.json — updated (new file)
✅ fixtures/expected/readme.txt — no changes
Acceptance Criteria
Summary
Add a
--update-snapshotsflag towaza runthat updates diff grader baseline files when they've drifted, similar to Jest's--updateSnapshotor Go's-updateflag.Motivation
The
diff_graderalready compares workspace files against expected snapshots. But when intentional changes are made, manually updating all snapshot files is tedious. An auto-update mode captures the current output as the new baseline.Proposed Implementation
Flag:
waza run eval.yaml --update-snapshotsWhen enabled:
Output
Acceptance Criteria
--update-snapshotsflag onwaza run