.NET: Add a verify-samples tool and skill#5005
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new .NET CLI tool under dotnet/samples/verify-samples/ to run sample projects and validate their output (deterministic substring checks + AI/structured-output verification for nondeterministic samples), plus a GitHub skill doc describing how to use and maintain it.
Changes:
- Introduces the
verify-samplesrunner/orchestrator, output verifier, and CSV/log writers. - Adds curated sample-definition registries for
01-get-started,02-agents, and03-workflows. - Wires the tool into
dotnet/agent-framework-dotnet.slnxand documents usage via a new skill.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| dotnet/samples/verify-samples/Program.cs | CLI entrypoint: resolves repo root, sets up AI verifier client, runs orchestrator, writes summaries. |
| dotnet/samples/verify-samples/VerifyOptions.cs | Parses CLI args, registers sample categories, builds filtered sample list. |
| dotnet/samples/verify-samples/VerificationOrchestrator.cs | Filters skipped/runnable samples, runs samples in parallel, reports/logs results. |
| dotnet/samples/verify-samples/SampleRunner.cs | Executes dotnet run for sample projects, captures stdout/stderr, supports stdin injection + timeout. |
| dotnet/samples/verify-samples/SampleVerifier.cs | Performs exit-code + substring checks; uses RunAsync<T> structured-output to verify nondeterministic output. |
| dotnet/samples/verify-samples/VerificationResult.cs | Result model used for console/log/CSV reporting. |
| dotnet/samples/verify-samples/ConsoleReporter.cs | Thread-safe, prefixed, colored console output + end-of-run summary. |
| dotnet/samples/verify-samples/LogFileWriter.cs | Thread-safe sequential log writer with per-sample output sections and final summary. |
| dotnet/samples/verify-samples/CsvResultWriter.cs | Writes a CSV summary of pass/fail/skip results. |
| dotnet/samples/verify-samples/SampleDefinition.cs | Defines per-sample config: paths, env vars, deterministic checks, AI expectations, stdin inputs. |
| dotnet/samples/verify-samples/GetStartedSamples.cs | Sample definitions for 01-get-started. |
| dotnet/samples/verify-samples/AgentsSamples.cs | Sample definitions for 02-agents. |
| dotnet/samples/verify-samples/WorkflowSamples.cs | Sample definitions for 03-workflows. |
| dotnet/samples/verify-samples/verify-samples.csproj | New executable project with dependencies + removal of inherited sample helpers. |
| dotnet/agent-framework-dotnet.slnx | Adds the verify-samples project to the solution. |
| dotnet/.github/skills/verify-samples-tool/SKILL.md | Documentation/skill: running the tool, categories, and authoring sample definitions. |
SergeyMenshykh
approved these changes
Apr 1, 2026
rogerbarreto
approved these changes
Apr 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
This tool can run samples and verify that they work correctly based on configurations that determine what inputs to pass to samples and what outputs to expect.
There is also a skill, which can be used to build/modify the configurations and run specific samples using the tool
Description
Contribution Checklist