[ty] Fix GitHub-annotations mdtest output format#23694
Merged
AlexWaygood merged 4 commits intomainfrom Mar 3, 2026
Merged
Conversation
…to stdout GitHub Actions workflow commands (like `::error`) must appear at the beginning of a line in stdout to be detected. The previous code collected `::error` lines in an assertion string and output them via `assert!()`, but `libtest_mimic` wraps panic messages with a `test panicked: ` prefix, which prevented GitHub from parsing the workflow commands. This restores the original behavior from #17150 by printing `::error` annotations directly to stdout using `println!()`, while still collecting CLI-format errors in the assertion string for the test failure message. https://claude.ai/code/session_01Pz2xREd4bteUUqxyg3QzpM
…nnotations This commit intentionally introduces wrong `revealed:` expectations to test that `::error` GitHub annotations render correctly on the PR diff. This commit should be reverted before merging. https://claude.ai/code/session_01Pz2xREd4bteUUqxyg3QzpM
…line on PR diffs GitHub requires file paths in `::error` workflow commands to be relative to the repository root for annotations to appear inline on the "Files changed" tab. The code was using the absolute filesystem path, which caused annotations to appear only in the Actions job log. https://claude.ai/code/session_01Pz2xREd4bteUUqxyg3QzpM
…GitHub annotations" This reverts commit 213ba68.
carljm
approved these changes
Mar 3, 2026
| for (relative_line_number, failures) in test_failures.by_line.iter() { | ||
| let file = match output_format { | ||
| OutputFormat::Cli => relative_fixture_path.as_str(), | ||
| OutputFormat::GitHub => absolute_fixture_path.as_str(), |
Contributor
There was a problem hiding this comment.
I guess using the absolute path for Github format isn't actually necessary? We stop doing it here and I don't see it moved anywhere else in the PR.
Member
Author
There was a problem hiding this comment.
It's not just unnecessary, it's incorrect. The first version of this PR didn't make this change, but I still didn't see any annotations showing up in the "files changed" tab. Claude diagnosed that the reason was that we were using absolute paths here rather than relative paths.
carljm
added a commit
that referenced
this pull request
Mar 3, 2026
* main: [ty] Apply narrowing to walrus values (#23687) [`perflint`] Extend `PERF102` to comprehensions and generators (#23473) [ty] Fix GitHub-annotations mdtest output format (#23694) [ty] Reduce the number of potentially-flaky projects (#23698) [`pydocstyle`] Fix numpy section ordering (`D420`) (#23685) [ty] Move method-related types to a submodule (#23691) [ty] Avoid the mandatory "ecosystem-analyzer workflow run cancelled" notification every time you make a PR (#23695) [ty] Move `Type::subtyping_is_always_reflexive` to `types::relation` (#23692) Update conformance suite commit hash (#23693) [ty] Add mdtest suite for `typing.Concatenate` (#23554) [ty] filter out pre-loop bindings from loop headers (#23536)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#22081 fixed one regression but introduced a new one. It fixed the way mdtest failures are reported for local use, so
mdtest.pyworks well, but it broke the GitHub-annotations mdtest format that I added way back in #17150. In order for GitHub annotations to work, errors must be printed to stdout. This PR fixes the regression, restoring the feature where mdtest failures would be beautifully rendered by GitHub annotations on the "files changed" tab.Test plan
An earlier commit on this PR branch broke some mdtests deliberately to test the feature. Screenshot of the "files changed" tab before that commit was reverted:
Screenshot
I also manually ran
mdtest.pylocally to check that the format there was still good.