feat: Snapshot auto-update workflow for diff grader by spboyer · Pull Request #95 · microsoft/waza

spboyer · 2026-03-05T02:31:03Z

Closes #85

codecov-commenter · 2026-03-05T02:35:04Z

Codecov Report

❌ Patch coverage is 61.08374% with 79 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@091a1f9). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
cmd/waza/cmd_run.go	22.80%	41 Missing and 3 partials ⚠️
internal/graders/diff_grader.go	79.83%	14 Missing and 11 partials ⚠️
internal/graders/grader.go	0.00%	6 Missing ⚠️
internal/orchestration/runner.go	75.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #95   +/-   ##
=======================================
  Coverage        ?   72.62%           
=======================================
  Files           ?      129           
  Lines           ?    14605           
  Branches        ?        0           
=======================================
  Hits            ?    10607           
  Misses          ?     3201           
  Partials        ?      797

Flag	Coverage Δ
go-implementation	`72.62% <61.08%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR implements the --update-snapshots flag for waza run, analogous to Jest's --updateSnapshot. When enabled, instead of failing on diff grader mismatches, the current workspace output is written back as the new snapshot baseline. It integrates the flag end-to-end: CLI flag → runner option → grader parameter → grader behavior → summary output.

Changes:

Adds UpdateSnapshots field to DiffGraderArgs/diffGrader and implements update/create logic in checkSnapshot, plus a countChangedLines helper
Adds WithUpdateSnapshots runner option that injects update_snapshots into diff grader params during runGraders
Adds --update-snapshots CLI flag, threads it through runSingleModel, and renders the update summary in printSnapshotUpdateSummary

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`internal/graders/diff_grader.go`	Core snapshot update logic: new `snapshotUpdate` struct, update/create paths in `checkSnapshot`, `writeSnapshot` helper, `countChangedLines` function, updated `buildResult` feedback
`internal/graders/diff_grader_test.go`	New test file covering mismatch-without-update (fail), update+create flow, and no-changes flow
`internal/orchestration/runner.go`	New `updateSnapshots` field on `TestRunner`, `WithUpdateSnapshots` option, injection of `update_snapshots` param into diff grader configs in `runGraders`
`internal/orchestration/runner_orchestration_test.go`	Integration test verifying the runner correctly updates snapshot files via `WithUpdateSnapshots`
`cmd/waza/cmd_run.go`	New `updateSnapshots` global flag, Cobra flag registration, `WithUpdateSnapshots` option passed to runner, `printSnapshotUpdateSummary` output function
`cmd/waza/cmd_run_test.go`	Flag-parsing test for `--update-snapshots` and reset in `resetRunGlobals`
`site/src/content/docs/reference/cli.mdx`	Adds `--update-snapshots` to the flags table and examples
`.squad/decisions.md`	Squad decision log entries for token diff strategy and model/workflow directive
`.squad/log/2026-03-05T00-36-issue-assignment-pipeline.md`	New session log file
`.squad/log/2026-03-05T00-26-rusty-token-diff-design.md`	New session log file

Comments suppressed due to low confidence (1)

internal/orchestration/runner.go:1227

Same direct mutation of vCfg.Parameters (via the params alias) issue as in the global validators loop above. When the diff grader parameters map is non-nil, params["update_snapshots"] = true mutates the original spec's parameter map rather than creating a copy, as injectJudgeModel does.

		if r.updateSnapshots && kind == models.GraderKindDiff {
			params["update_snapshots"] = true
		}

spboyer

LGTM — Rusty. Snapshot auto-update is well-designed. --update-snapshots flows cleanly through runner → diff grader. snapshotUpdate struct with create/update/unchanged states gives good UX feedback. countChangedLines is a nice touch. 3 grader tests + runner integration test = solid coverage. Ship it. (Self-authored PR — cannot self-approve via API.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

wbreza

Code Review: PR #95 - feat: Snapshot auto-update workflow for diff grader

What Looks Good

Path traversal protection via resolveSnapshotPath with symlink resolution
Exported SnapshotUpdate struct and status constants
parseSnapshotUpdates handles typed slices, pointer slices, and JSON fallback
Non-update mode behavior is unchanged
New snapshot creation when file doesn't exist + update mode
CRLF normalization in line diff counting

High (1 finding)

Params map mutation in runner.go - When vCfg.Parameters is non-nil, setting params["update_snapshots"] = true mutates the original spec grader config via reference alias. If the same spec is reused across multi-trial runs, snapshot updates would be enabled for all subsequent trials unintentionally. Verify whether the runner clones the params map or shares it.

Medium (2 findings)

snapshot_updates always in Details - Empty slice emitted even when not in update mode. Harmless but noisy.
countChangedLines is positional - Single line insertion reports all subsequent lines as changed. Acceptable for summary heuristic but may mislead.

Summary

Priority	Count
Critical	0
High	1
Medium	2
Low	0

Overall Assessment: Comment - solid implementation with good security (path traversal protection). The params map mutation concern should be verified before approving.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Copilot AI review requested due to automatic review settings March 5, 2026 02:31

spboyer requested review from chlowell and richardpark-msft as code owners March 5, 2026 02:31

spboyer self-assigned this Mar 5, 2026

github-actions Bot enabled auto-merge (squash) March 5, 2026 02:31

Copilot started reviewing on behalf of spboyer March 5, 2026 02:32 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Comment thread internal/graders/diff_grader.go Outdated

Comment thread internal/orchestration/runner.go

Comment thread site/src/content/docs/reference/cli.mdx Outdated

Comment thread cmd/waza/cmd_run.go Outdated

spboyer commented Mar 5, 2026

View reviewed changes

spboyer force-pushed the squad/85-snapshot-auto-update branch from e019d5a to 7b76de9 Compare March 5, 2026 17:12

spboyer added a commit that referenced this pull request Mar 5, 2026

fix: address review feedback on PR #95

8df06b1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 5, 2026 17:36

spboyer added a commit that referenced this pull request Mar 5, 2026

fix: address review feedback on PR #95

03e93a5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

spboyer force-pushed the squad/85-snapshot-auto-update branch from 8df06b1 to 03e93a5 Compare March 5, 2026 17:47

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Comment thread internal/graders/diff_grader.go Outdated

Comment thread internal/graders/diff_grader.go

Comment thread cmd/waza/cmd_run.go

spboyer added a commit that referenced this pull request Mar 5, 2026

fix: address PR #95 diff snapshot review issues

5f73d8f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 5, 2026 20:40

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Comment thread internal/graders/diff_grader.go

Comment thread cmd/waza/cmd_run.go Outdated

Comment thread internal/graders/diff_grader.go

Comment thread internal/graders/diff_grader_test.go

chlowell pushed a commit to chlowell/waza that referenced this pull request Mar 5, 2026

docs: update README for Go CLI, remove Python refs (microsoft#95)

15a26e4

spboyer and others added 5 commits March 5, 2026 19:04

feat: snapshot auto-update workflow for diff grader microsoft#85

49d9990

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: address review feedback on PR microsoft#95

b9e9c61

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: address PR microsoft#95 diff snapshot review issues

b1c9e54

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: gofmt cmd_run_test.go

807e788

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: address snapshot review feedback in PR95

3963ee3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 6, 2026 00:04

spboyer force-pushed the squad/85-snapshot-auto-update branch from 5bc4c57 to 3963ee3 Compare March 6, 2026 00:04

Copilot started reviewing on behalf of spboyer March 6, 2026 00:06 View session

wbreza reviewed Mar 6, 2026

View reviewed changes

Copilot AI reviewed Mar 6, 2026

View reviewed changes

Comment thread internal/graders/diff_grader.go

Comment thread internal/graders/diff_grader.go

Comment thread internal/graders/diff_grader.go

Comment thread internal/graders/diff_grader_test.go

docs: Add comment to countChangedLines explaining diff limitation

6a69078

chlowell approved these changes Mar 9, 2026

View reviewed changes

github-actions Bot merged commit 9ed2df1 into microsoft:main Mar 9, 2026
6 checks passed

spboyer mentioned this pull request Mar 12, 2026

Release v0.21.0 #122

Merged

4 tasks

spboyer mentioned this pull request Apr 22, 2026

feat: Snapshot auto-update workflow for diff grader #85

Closed

4 tasks

Uh oh!

Conversation

spboyer commented Mar 5, 2026

Uh oh!

codecov-commenter commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spboyer left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wbreza left a comment

Choose a reason for hiding this comment

Code Review: PR #95 - feat: Snapshot auto-update workflow for diff grader

What Looks Good

High (1 finding)

Medium (2 findings)

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Mar 5, 2026 •

edited

Loading