Skip to content

Auto-trigger mypy primer classifier #2568

Closed
migeed-z wants to merge 9 commits intomainfrom
two-pass-llm-classification
Closed

Auto-trigger mypy primer classifier #2568
migeed-z wants to merge 9 commits intomainfrom
two-pass-llm-classification

Conversation

@migeed-z
Copy link
Contributor

@migeed-z migeed-z commented Feb 26, 2026

Tested the tool on some PRs manually.
Example run: #2567

I would like to gather some feedback on the took before tuning it further, so I am proposing to make the workflow automatic.

I added a feedback thumbs up/thumbs down to gather feedback in the last diff. I increased the font size to make it more visible.

…ifier

Split LLM classification into two passes to fix verdict-reasoning
contradictions (4/26 in PR #2493). Pass 1 produces reasoning and
PR attribution without a verdict. Pass 2 reads the reasoning and
assigns the verdict. This separates code analysis (hard) from
labeling (easy), eliminating cases where the LLM commits to a
verdict early and writes contradictory reasoning.

Also adds --pyrefly-diff CLI flag to include the pyrefly PR code
diff in each LLM call, enabling per-project attribution of which
code change caused errors to appear or disappear.
Restructure format_markdown() to show an overview table with linked
function names and file paths, collapsible detailed analysis, and a
suggested fix section. Add helpers for function-name linkification
and root cause extraction from PR attribution text.
Add --suggest CLI flag, Suggestion/SuggestionResult dataclasses, and
generate_suggestions() LLM client that produces actionable source code
fix suggestions from classification results and the PR diff.
Use a stricter regex (_INTERNAL_FUNCTION_PATTERN) that requires
underscores to distinguish pyrefly internal function names like
check_for_imported_final_reassignment() from common Python method
names like get(), match(), set() that appear in error messages.
Asks reviewers to react with 👍 or 👎 so we can track
classifier accuracy over time.
…e display

- Collect error_kinds from both project.added and project.removed so
  improvement-only diffs (all removals) still populate the Error Kinds column
- Add file path fallback in _extract_root_cause when no function name is found
- Add high-level summary paragraph aggregating patterns across projects
- Rename table header from "Error Kind" to "Error Kinds"
…lity

- Change workflow trigger from workflow_dispatch to workflow_run so it
  runs automatically when "Run mypy_primer" completes
- Move feedback prompt out of <sub> tag for better visibility
@meta-cla meta-cla bot added the cla signed label Feb 26, 2026
@migeed-z migeed-z marked this pull request as draft February 26, 2026 18:47
@migeed-z migeed-z changed the title Enhance primer classifier with attribution, suggestions, and auto-trigger Auto-trigger mypy primer classifier Feb 26, 2026
@meta-codesync
Copy link

meta-codesync bot commented Feb 26, 2026

@migeed-z has imported this pull request. If you are a Meta employee, you can view this in D94549072.

@migeed-z migeed-z force-pushed the two-pass-llm-classification branch from b135591 to 01c3601 Compare February 27, 2026 05:46
@migeed-z migeed-z force-pushed the two-pass-llm-classification branch 3 times, most recently from 48fa87d to 9dd6c3a Compare February 27, 2026 17:48
- Add --pr-description CLI flag to pass PR title/body to the classifier
- Include PR description in the LLM user prompt as "author's stated intent"
- Add INTENTIONAL REGRESSIONS guidance to system prompt so the LLM
  distinguishes conformance-driven regressions from unintentional bugs
- Update workflow to fetch PR title/body and pass it to the classifier
@migeed-z migeed-z force-pushed the two-pass-llm-classification branch from 9dd6c3a to eef5ab5 Compare February 27, 2026 17:53
Copy link
Contributor

@grievejia grievejia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.

@meta-codesync
Copy link

meta-codesync bot commented Feb 28, 2026

@migeed-z merged this pull request in b6b8737.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants