feat: enhance PR template validation with semantic checks#30541
Conversation
This update introduces a set of semantic validators for the PR template, ensuring that all required sections are filled out correctly before a PR can be marked as "Ready for review." The new checks include validation for the description, changelog entry, related issues, manual testing steps, and screenshots. Additionally, a sticky comment will be added to PRs that do not meet the criteria, providing feedback to authors on what needs to be addressed. This change aims to improve the quality and completeness of PR submissions.
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
|
There was a problem hiding this comment.
Pull request overview
This PR enhances the existing check-template-and-add-labels GitHub Action by adding semantic validation of PR template completeness (aligned with docs/readme/ready-for-review.md) and providing consolidated author feedback via a single sticky bot comment, while making the check informational for draft PRs and blocking for non-drafts.
Changes:
- Expanded workflow triggers so the check updates immediately when PR draft state changes or new commits are pushed.
- Added a semantic PR-template validator module (aggregating failures in one pass) plus Jest tests for the validators.
- Added a sticky-comment helper to create/update/delete one consolidated bot comment listing all missing template items, and wired this into
check-template-and-add-labels.ts.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/check-template-and-add-labels.yml | Adds ready_for_review, converted_to_draft, and synchronize triggers for more responsive PR checks. |
| .github/scripts/shared/pr-template-comment.ts | Introduces sticky PR comment create/update/delete logic for aggregated template failures. |
| .github/scripts/shared/pr-template-checks.ts | Adds pure semantic validators + section extraction helpers and an aggregated runAllChecks. |
| .github/scripts/shared/pr-template-checks.test.ts | Adds unit tests covering the new semantic validators and helpers. |
| .github/scripts/check-template-and-add-labels.ts | Integrates semantic checks + sticky comment behavior and draft-aware pass/fail semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Copilot identified that the previous cursor update (open + 4) when no closing --> is found left the remainder of the unclosed comment body in the output, allowing template placeholders inside a malformed comment to reach the semantic validators as real content. Replaced with an explicit break, which drops everything from the opener to the end of the string. Added a regression test to pin this behaviour.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85e3461b6d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…hecks - hasValidIssueLink: switch from .match() to matchAll() so a blank template `Fixes:` line no longer shadows a valid `Refs:` added below it. Also tighten `\s*` to `[ \t]*` after the colon to prevent the match from crossing a line boundary. - hasAllAuthorChecklistChecked: require at least one checked box so a section where all rows were deleted does not silently pass. - Add tests covering both new behaviours.
…dators - Introduce shared/markdown-tokenizer.ts: a zero-dependency, state-machine tokenizer that correctly handles fenced code blocks, HTML comments, headings, and GFM checkboxes in precedence order, preventing content injection from fenced blocks into validator logic - Migrate pr-template-checks.ts validators to use the tokenizer instead of regex-based string parsing - Fix TypeScript CFA narrowing error by changing scanLine to return ParseState directly rather than via a callback closure (closure assignments are not tracked in loop-join analysis) - Add unit tests and READMEs for markdown-tokenizer, pr-template-checks, and pr-template-comment - Consolidate PR section title source of truth into pr-template-checks.ts
Updated the error message in the PR template validation to point to the correct line in `pr-template-checks.ts`, ensuring accurate guidance for users when section titles are missing in PR bodies.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd0cf30053
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Added `mms-check` directives to the pull request template to enforce validation rules for each section. This includes checks for required content, changelog entries, issue links, manual testing steps, screenshots, and checklist completion. Updated the validation logic in `pr-template-checks.ts` to parse these directives and ensure compliance during PR submissions. Enhanced documentation to reflect these changes and provide guidance on the new validation system.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a020f8c. Configure here.
| fenceStart = -1; | ||
| fenceChar = ''; | ||
| fenceLen = 0; | ||
| } |
There was a problem hiding this comment.
Fence close regex mismatches tokenizer
Medium Severity
In computeFencedRanges, any line starting with a fence run is treated as a closing fence, including lines like ```javascript that still have text after the backticks. The markdown tokenizer only closes on lines that are fence markers alone, so stripHtmlComments can end a code block early and mis-strip HTML comments in the rest of a section.
Reviewed by Cursor Bugbot for commit a020f8c. Configure here.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #30541 +/- ##
==========================================
+ Coverage 82.86% 82.99% +0.13%
==========================================
Files 5582 5605 +23
Lines 144198 144358 +160
Branches 33521 33533 +12
==========================================
+ Hits 119483 119814 +331
+ Misses 16682 16480 -202
- Partials 8033 8064 +31 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|





Description
Extends the
check-template-and-add-labelsCI job with semantic validators that enforce the Definition of Ready For Review (docs/readme/ready-for-review.md) at the PR level, replacing the previous structural-only check.What changed:
shared/markdown-tokenizer.ts— a zero-dependency, single-pass GFM tokenizer (state machine) that correctly handles fenced code blocks, HTML comments, ATX headings, and GFM checkboxes in precedence order. This prevents content injection: aFixes: #123line or aCHANGELOG entry:inside a Gherkin block can no longer bypass the semantic validators.shared/pr-template-checks.tswith six pure validators (description non-empty, changelog entry present, issue linkage, manual testing steps filled, screenshots/evidence provided, author checklist fully checked). The validation plan is now metadata-driven:mms-checkdirectives embedded inpull-request-template.mddeclare which validator applies to each section; abuildValidationPlanparser reads them at module load and dispatches each section to aVALIDATORSregistry. Adding or reordering a plain-text section needs zero code changes — only a new kind of check requires a new registry entry.PR_TEMPLATE_SECTION_TITLES(structural presence list) andAUTHOR_CHECKLIST_MIN_ITEMS(minimum required checklist rows) are now derived at module load from the same template file — no hardcoded constants. The checklist count enforces that authors cannot bypass the requirement by deleting unchecked rows.pull-request-template.mdis the single source of truth for validation rules. It now embeds<!-- mms-check: type=X required=true -->directives (invisible in rendered markdown) directly under each author-validated section heading, with a legend comment at the top explaining the vocabulary.shared/pr-template-comment.tsthat creates, updates, or deletes a single sticky bot comment listing every missing item, so authors get consolidated actionable feedback.check-template-and-add-labels.tsto call the validators, apply draft-aware exit (informationalcore.warning+ exit 0 for drafts, blockingcore.setFailed+ exit 1 for non-drafts), and resolve the January-2024 TODO that kept the structural mismatch non-failing.ready_for_review,converted_to_draft, andsynchronizeso the required status flips immediately when an author changes draft state or pushes new commits.markdown-tokenizer,pr-template-checks, andpr-template-commentdocumenting API, parsing workflow (Mermaid diagram), directive vocabulary, and how to add new validators.Changelog
CHANGELOG entry: null
Related issues
Refs: MCWP-603
Manual testing steps
Local validator script (copy and run in your local terminal, at project root)
Screenshots/Recordings
Before
N/A
After
Proof of manual testing passed:
Example of sticky PR comment the check would send on a PR marked ready for review but with no changelog line:
Pre-merge author checklist
Performance checks (if applicable)
trace()for usage andaddTokenfor an exampleFor performance guidelines and tooling, see the Performance Guide.
Pre-merge reviewer checklist
Note
Medium Risk
Changes merge-blocking CI behavior and
pull_request_targetcheckout semantics; incorrect validation or draft/ready flips could block or mis-label many PRs, though rules are read from the protected base-branch template.Overview
This PR turns the Check template and add labels workflow from “section headings present + changelog line” into Definition of Ready for Review enforcement: description, changelog, issue link, manual testing, screenshots, and author checklist must be materially filled, not just titled.
pull-request-template.mdnow carries invisible<!-- mms-check: type=… required=… -->directives;pr-template-checks.tsreads the base-branch template at module load, builds a validation plan, and runs six semantic validators. Section lists and minimum checklist row count are derived from that template instead of hardcoded arrays intemplate.ts.A new
markdown-tokenizerparses PR bodies (fences, comments, headings, checkboxes) so placeholders inside Gherkin/fences cannot satisfy changelog,Fixes:/Refs:, or checklist rules. Failures are aggregated into one sticky bot comment (pr-template-comment); drafts getcore.warningand exit 0, ready-for-review PRscore.setFailedand exit 1. The old transitional “structural mismatch does not fail” behavior is removed.The workflow adds
ready_for_review,converted_to_draft, andsynchronizetriggers; checkout stays on the default ref (documented: noref:onpull_request_target). Broad Jest coverage documents injection and directive parsing.Reviewed by Cursor Bugbot for commit a020f8c. Bugbot is set up for automated code reviews on this repo. Configure here.