Instructions for updating VerifyDiagnostics and VerifyIL expectations in tests#83067
Merged
jcouv merged 1 commit intodotnet:mainfrom Apr 8, 2026
Merged
Instructions for updating VerifyDiagnostics and VerifyIL expectations in tests#83067jcouv merged 1 commit intodotnet:mainfrom
jcouv merged 1 commit intodotnet:mainfrom
Conversation
333fred
reviewed
Apr 7, 2026
| - **Unit tests**: Test individual compiler phases (lexing, parsing) | ||
| - **Compilation tests**: Create `Compilation` objects and verify symbols/diagnostics | ||
| - **Cross-language patterns**: Many test patterns work for both C# and VB with minor syntax changes | ||
| - **Verification baselines**: When helpers like `VerifyDiagnostics`, `VerifyEmitDiagnostics`, `VerifyIL`, and similar compiler test APIs fail with an `Actual:` block containing the expected test content, copy that block directly into the verification call. |
Member
There was a problem hiding this comment.
Do comments like "align with existing baseline texts" help? Should we make this a standalone skill?
Member
Author
There was a problem hiding this comment.
I tried adding that sentence to instructions, but I didn't notice a difference.
The main problem I was trying to solve was the model putting stuff like:
CreateCompilation(source, options: TestOptions.UnsafeDebugExe).VerifyDiagnostics(
Diagnostic(ErrorCode.WRN_AddressOfInAsync, "localFuncLocal").WithLocation(32, 39));
With the instructions, it correctly copied the comments in a few cases that I tried, so I'm happy.
FWIW, asked the model to reflect on that and it concurred :-P (copied below).
I'm wondering whether including "align with existing baseline texts" in the instructions for `VerifyDiagnostics` was helpful. What do you think?
● Looking at how I solved this — I ran the test, copied the Actual: block from the failure output directly into the VerifyDiagnostics call. That's exactly the standard workflow documented in the repo's custom instructions ("copy that block directly into the verification call").
I didn't need any "align with existing baseline texts" guidance to do that. The existing instruction about copying the Actual: block was sufficient. An extra instruction about aligning with baseline texts could actually be counterproductive — it might encourage trying to
manually massage the expected output rather than just trusting the compiler's actual output.
So no, I don't think it added value here. The simpler, existing guidance was enough.
As for skill, maybe at some point. The advantage of a skill is that we could go into more details, include examples and such, without cluttering the context window by default. But so far, I found this one line instruction to be effective, so seems fine as first step.
333fred
approved these changes
Apr 8, 2026
This was referenced Apr 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Microsoft Reviewers: Open in CodeFlow