Skip to content

Test tooling: Use stricter regex for lines in expected_results.txt files, provide readable errors for format errors#5948

Merged
StephanTLavavej merged 5 commits intomicrosoft:mainfrom
vmichal:report-incorrect-format-expected_results
Dec 11, 2025
Merged

Test tooling: Use stricter regex for lines in expected_results.txt files, provide readable errors for format errors#5948
StephanTLavavej merged 5 commits intomicrosoft:mainfrom
vmichal:report-incorrect-format-expected_results

Conversation

@vmichal
Copy link
Contributor

@vmichal vmichal commented Dec 7, 2025 •
edited
Loading

When parsing expected_results.txt for some test suite, the python script was not ready to handle the following issues:

  • Missing result code (only file name present) yielded generic "AttributeError: 'NoneType' object has no attribute 'group'" when attempting to access regex match groups.
  • Typo in result code yielded "AttributeError: module 'stl.test.tests' has no attribute", which is arguably better, but still could be better.

Both issues are quite easy to trigger when experimenting with a larger number of tests, so user-friendly error messages seem like a nice touch.

@vmichal vmichal requested a review from a team as a code owner December 7, 2025 23:32
@github-project-automation github-project-automation bot moved this to Initial Review in STL Code Reviews Dec 7, 2025
@StephanTLavavej StephanTLavavej added the test Related to test code label Dec 8, 2025
@StephanTLavavej StephanTLavavej self-assigned this Dec 8, 2025
@vmichal
Copy link
Contributor Author

vmichal commented Dec 8, 2025 •
edited
Loading

BTW I am more than open to alternative wording of error messages. I experimented with a few alternatives and this one I disliked the least.

StephanTLavavej
StephanTLavavej reviewed Dec 10, 2025
View reviewed changes
@StephanTLavavej
Copy link
Member

StephanTLavavej commented Dec 10, 2025

Thanks! 😻 After my commits, the messages are:

Exception: Unknown result code "WOOF" in line "std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique.array.pass.cpp:0 WOOF" in D:/GitHub/STL/tests/libcxx/expected_results.txt.
Exception: Incorrectly formatted line "std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique.array.pass.cpp:0" in D:/GitHub/STL/tests/libcxx/expected_results.txt.
Exception: Incorrectly formatted line "std/utilities/smartptr/unique.ptr/unique.ptr.create/make_unique.array.pass.cpp:0 FA:IL" in D:/GitHub/STL/tests/libcxx/expected_results.txt.

We could put additional effort into special-casing an exception for a missing result code (by far my most common mistake) but this is already such an improvement over the status quo.

@StephanTLavavej StephanTLavavej removed their assignment Dec 10, 2025
@StephanTLavavej StephanTLavavej moved this from Initial Review to Ready To Merge in STL Code Reviews Dec 10, 2025
@StephanTLavavej StephanTLavavej moved this from Ready To Merge to Merging in STL Code Reviews Dec 11, 2025
@StephanTLavavej
Copy link
Member

StephanTLavavej commented Dec 11, 2025

I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed.

@StephanTLavavej StephanTLavavej merged commit da23b20 into microsoft:main Dec 11, 2025
45 checks passed
@github-project-automation github-project-automation bot moved this from Merging to Done in STL Code Reviews Dec 11, 2025
@StephanTLavavej
Copy link
Member

StephanTLavavej commented Dec 11, 2025

Thanks for improving this easily-overlooked pain point for contributors! 😻 ✅ 🎉

@vmichal vmichal mentioned this pull request Jan 9, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@StephanTLavavej StephanTLavavej StephanTLavavej approved these changes

Assignees

No one assigned

Labels

test Related to test code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vmichal @StephanTLavavej