Skip to content

feat: enhance PR template validation with semantic checks#30541

Merged
NicolasMassart merged 13 commits into
mainfrom
refactor/MCWP-603_pr-template_part-2
Jun 4, 2026
Merged

feat: enhance PR template validation with semantic checks#30541
NicolasMassart merged 13 commits into
mainfrom
refactor/MCWP-603_pr-template_part-2

Conversation

@NicolasMassart

@NicolasMassart NicolasMassart commented May 21, 2026

Copy link
Copy Markdown
Contributor

Description

Extends the check-template-and-add-labels CI job with semantic validators that enforce the Definition of Ready For Review (docs/readme/ready-for-review.md) at the PR level, replacing the previous structural-only check.

What changed:

  • New module shared/markdown-tokenizer.ts — a zero-dependency, single-pass GFM tokenizer (state machine) that correctly handles fenced code blocks, HTML comments, ATX headings, and GFM checkboxes in precedence order. This prevents content injection: a Fixes: #123 line or a CHANGELOG entry: inside a Gherkin block can no longer bypass the semantic validators.
  • New module shared/pr-template-checks.ts with six pure validators (description non-empty, changelog entry present, issue linkage, manual testing steps filled, screenshots/evidence provided, author checklist fully checked). The validation plan is now metadata-driven: mms-check directives embedded in pull-request-template.md declare which validator applies to each section; a buildValidationPlan parser reads them at module load and dispatches each section to a VALIDATORS registry. Adding or reordering a plain-text section needs zero code changes — only a new kind of check requires a new registry entry.
  • PR_TEMPLATE_SECTION_TITLES (structural presence list) and AUTHOR_CHECKLIST_MIN_ITEMS (minimum required checklist rows) are now derived at module load from the same template file — no hardcoded constants. The checklist count enforces that authors cannot bypass the requirement by deleting unchecked rows.
  • pull-request-template.md is the single source of truth for validation rules. It now embeds <!-- mms-check: type=X required=true --> directives (invisible in rendered markdown) directly under each author-validated section heading, with a legend comment at the top explaining the vocabulary.
  • New module shared/pr-template-comment.ts that creates, updates, or deletes a single sticky bot comment listing every missing item, so authors get consolidated actionable feedback.
  • Updated check-template-and-add-labels.ts to call the validators, apply draft-aware exit (informational core.warning + exit 0 for drafts, blocking core.setFailed + exit 1 for non-drafts), and resolve the January-2024 TODO that kept the structural mismatch non-failing.
  • Extended workflow triggers with ready_for_review, converted_to_draft, and synchronize so the required status flips immediately when an author changes draft state or pushes new commits.
  • READMEs added for markdown-tokenizer, pr-template-checks, and pr-template-comment documenting API, parsing workflow (Mermaid diagram), directive vocabulary, and how to add new validators.

Changelog

CHANGELOG entry: null

Related issues

Refs: MCWP-603

Manual testing steps

Feature: semantic PR template validators

  Scenario: validators pass on a complete PR body
    Background: `gh` CLI authenticated (`gh auth status`) and the repo checked out on this branch.

    Given the branch is checked out locally
    When the reviewer runs the local validator script (see code below) against this PR's body
    Then the script exits 0 and prints "All checks pass"

Local validator script (copy and run in your local terminal, at project root)

gh pr view 30541 --json body -q .body > /tmp/pr-body.txt
cat > .github/scripts/check-pr-local.ts << 'EOF'
import { runAllChecks } from './shared/pr-template-checks';
import * as fs from 'fs';
const body = fs.readFileSync('/tmp/pr-body.txt', 'utf8');
const failures = runAllChecks(body, false);
if (failures.length === 0) { console.log('All checks pass'); }
else { failures.forEach((f: { reason: string }) => console.log('FAIL', f.reason)); process.exit(1); }
EOF
cd .github/scripts && yarn ts-node check-pr-local.ts ; rm check-pr-local.ts

Screenshots/Recordings

Before

N/A

After

Proof of manual testing passed:

image

Example of sticky PR comment the check would send on a PR marked ready for review but with no changelog line:

PR template — items to address before "Ready for review"

This check is blocking. Address every item below, then push to re-run.

  • Changelog section is missing the CHANGELOG entry: line. Add CHANGELOG entry: <description>, CHANGELOG entry: null, or apply the no-changelog label.

See docs/readme/ready-for-review.md for the full Definition of Ready for Review.

Pre-merge author checklist

Performance checks (if applicable)

  • I've tested on Android
    • Ideally on a mid-range device; emulator is acceptable
  • I've tested with a power user scenario
    • Use these power-user SRPs to import wallets with many accounts and tokens
  • I've instrumented key operations with Sentry traces for production performance metrics

For performance guidelines and tooling, see the Performance Guide.

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
Changes merge-blocking CI behavior and pull_request_target checkout semantics; incorrect validation or draft/ready flips could block or mis-label many PRs, though rules are read from the protected base-branch template.

Overview
This PR turns the Check template and add labels workflow from “section headings present + changelog line” into Definition of Ready for Review enforcement: description, changelog, issue link, manual testing, screenshots, and author checklist must be materially filled, not just titled.

pull-request-template.md now carries invisible <!-- mms-check: type=… required=… --> directives; pr-template-checks.ts reads the base-branch template at module load, builds a validation plan, and runs six semantic validators. Section lists and minimum checklist row count are derived from that template instead of hardcoded arrays in template.ts.

A new markdown-tokenizer parses PR bodies (fences, comments, headings, checkboxes) so placeholders inside Gherkin/fences cannot satisfy changelog, Fixes:/Refs:, or checklist rules. Failures are aggregated into one sticky bot comment (pr-template-comment); drafts get core.warning and exit 0, ready-for-review PRs core.setFailed and exit 1. The old transitional “structural mismatch does not fail” behavior is removed.

The workflow adds ready_for_review, converted_to_draft, and synchronize triggers; checkout stays on the default ref (documented: no ref: on pull_request_target). Broad Jest coverage documents injection and directive parsing.

Reviewed by Cursor Bugbot for commit a020f8c. Bugbot is set up for automated code reviews on this repo. Configure here.

This update introduces a set of semantic validators for the PR template, ensuring that all required sections are filled out correctly before a PR can be marked as "Ready for review." The new checks include validation for the description, changelog entry, related issues, manual testing steps, and screenshots. Additionally, a sticky comment will be added to PRs that do not meet the criteria, providing feedback to authors on what needs to be addressed. This change aims to improve the quality and completeness of PR submissions.
@github-actions

Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@metamaskbotv2 metamaskbotv2 Bot added the team-mobile-platform Mobile Platform team label May 21, 2026
Comment thread .github/scripts/shared/pr-template-checks.ts Fixed
@sonarqubecloud

Copy link
Copy Markdown

Comment thread .github/scripts/shared/pr-template-checks.ts Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the existing check-template-and-add-labels GitHub Action by adding semantic validation of PR template completeness (aligned with docs/readme/ready-for-review.md) and providing consolidated author feedback via a single sticky bot comment, while making the check informational for draft PRs and blocking for non-drafts.

Changes:

  • Expanded workflow triggers so the check updates immediately when PR draft state changes or new commits are pushed.
  • Added a semantic PR-template validator module (aggregating failures in one pass) plus Jest tests for the validators.
  • Added a sticky-comment helper to create/update/delete one consolidated bot comment listing all missing template items, and wired this into check-template-and-add-labels.ts.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
.github/workflows/check-template-and-add-labels.yml Adds ready_for_review, converted_to_draft, and synchronize triggers for more responsive PR checks.
.github/scripts/shared/pr-template-comment.ts Introduces sticky PR comment create/update/delete logic for aggregated template failures.
.github/scripts/shared/pr-template-checks.ts Adds pure semantic validators + section extraction helpers and an aggregated runAllChecks.
.github/scripts/shared/pr-template-checks.test.ts Adds unit tests covering the new semantic validators and helpers.
.github/scripts/check-template-and-add-labels.ts Integrates semantic checks + sticky comment behavior and draft-aware pass/fail semantics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/scripts/shared/pr-template-checks.ts
Comment thread .github/scripts/shared/pr-template-checks.test.ts
NicolasMassart and others added 2 commits June 3, 2026 18:07
Copilot identified that the previous cursor update (open + 4) when no
closing --> is found left the remainder of the unclosed comment body
in the output, allowing template placeholders inside a malformed comment
to reach the semantic validators as real content.

Replaced with an explicit break, which drops everything from the opener
to the end of the string. Added a regression test to pin this behaviour.
@NicolasMassart NicolasMassart marked this pull request as ready for review June 3, 2026 16:48
@NicolasMassart NicolasMassart requested a review from a team as a code owner June 3, 2026 16:48
Comment thread .github/scripts/shared/pr-template-checks.ts
@github-actions github-actions Bot added the risk:medium AI analysis: medium risk label Jun 3, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85e3461b6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/shared/pr-template-checks.ts Outdated
…hecks

- hasValidIssueLink: switch from .match() to matchAll() so a blank
  template `Fixes:` line no longer shadows a valid `Refs:` added below
  it. Also tighten `\s*` to `[ \t]*` after the colon to prevent the
  match from crossing a line boundary.
- hasAllAuthorChecklistChecked: require at least one checked box so a
  section where all rows were deleted does not silently pass.
- Add tests covering both new behaviours.
Comment thread .github/scripts/shared/pr-template-checks.ts
@NicolasMassart NicolasMassart marked this pull request as draft June 4, 2026 08:59
…dators

- Introduce shared/markdown-tokenizer.ts: a zero-dependency, state-machine
  tokenizer that correctly handles fenced code blocks, HTML comments, headings,
  and GFM checkboxes in precedence order, preventing content injection from
  fenced blocks into validator logic
- Migrate pr-template-checks.ts validators to use the tokenizer instead of
  regex-based string parsing
- Fix TypeScript CFA narrowing error by changing scanLine to return ParseState
  directly rather than via a callback closure (closure assignments are not
  tracked in loop-join analysis)
- Add unit tests and READMEs for markdown-tokenizer, pr-template-checks, and
  pr-template-comment
- Consolidate PR section title source of truth into pr-template-checks.ts
@github-actions github-actions Bot added size-XL and removed size-L labels Jun 4, 2026
@NicolasMassart NicolasMassart requested a review from Copilot June 4, 2026 11:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Comment thread .github/scripts/check-template-and-add-labels.ts
NicolasMassart and others added 2 commits June 4, 2026 13:45
Updated the error message in the PR template validation to point to the correct line in `pr-template-checks.ts`, ensuring accurate guidance for users when section titles are missing in PR bodies.
@NicolasMassart NicolasMassart marked this pull request as ready for review June 4, 2026 11:48

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd0cf30053

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/shared/pr-template-checks.ts Outdated
Comment thread .github/scripts/shared/pr-template-checks.ts
@NicolasMassart NicolasMassart marked this pull request as draft June 4, 2026 12:29
NicolasMassart and others added 2 commits June 4, 2026 15:31
Added `mms-check` directives to the pull request template to enforce validation rules for each section. This includes checks for required content, changelog entries, issue links, manual testing steps, screenshots, and checklist completion. Updated the validation logic in `pr-template-checks.ts` to parse these directives and ensure compliance during PR submissions. Enhanced documentation to reflect these changes and provide guidance on the new validation system.
@NicolasMassart NicolasMassart marked this pull request as ready for review June 4, 2026 14:47
@NicolasMassart NicolasMassart added area-devex Issues and PRs focused on developer experience skip-e2e skip E2E test jobs github_actions Pull requests that update GitHub Actions code labels Jun 4, 2026

@Cal-L Cal-L left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-project-automation github-project-automation Bot moved this from Needs dev review to Review finalised - Ready to be merged in PR review queue Jun 4, 2026
@NicolasMassart NicolasMassart enabled auto-merge June 4, 2026 17:22

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a020f8c. Configure here.

fenceStart = -1;
fenceChar = '';
fenceLen = 0;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fence close regex mismatches tokenizer

Medium Severity

In computeFencedRanges, any line starting with a fence run is treated as a closing fence, including lines like ```javascript that still have text after the backticks. The markdown tokenizer only closes on lines that are fence markers alone, so stripHtmlComments can end a code block early and mis-strip HTML comments in the rest of a section.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a020f8c. Configure here.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.99%. Comparing base (000d79f) to head (a020f8c).
⚠️ Report is 92 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #30541      +/-   ##
==========================================
+ Coverage   82.86%   82.99%   +0.13%     
==========================================
  Files        5582     5605      +23     
  Lines      144198   144358     +160     
  Branches    33521    33533      +12     
==========================================
+ Hits       119483   119814     +331     
+ Misses      16682    16480     -202     
- Partials     8033     8064      +31     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@NicolasMassart NicolasMassart added this pull request to the merge queue Jun 4, 2026
Merged via the queue into main with commit dc8924e Jun 4, 2026
106 of 109 checks passed
@NicolasMassart NicolasMassart deleted the refactor/MCWP-603_pr-template_part-2 branch June 4, 2026 18:04
@github-project-automation github-project-automation Bot moved this from Review finalised - Ready to be merged to Merged, Closed or Archived in PR review queue Jun 4, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 4, 2026
@metamaskbotv2 metamaskbotv2 Bot added the release-7.81.0 Issue or pull request that will be included in release 7.81.0 label Jun 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-devex Issues and PRs focused on developer experience github_actions Pull requests that update GitHub Actions code release-7.81.0 Issue or pull request that will be included in release 7.81.0 risk:medium AI analysis: medium risk size-XL skip-e2e skip E2E test jobs team-mobile-platform Mobile Platform team

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

5 participants