test: integrate AI test plan into RC build comments by sleepytanya · Pull Request #28951 · MetaMask/metamask-mobile

sleepytanya · 2026-04-16T21:05:58Z

Description

Integrates AI test plan generation directly into RC build comments

Changes

New scripts/build-announce/ module (TypeScript):
- index.ts - Main orchestrator (replaces post-rc-build-comment.mjs)
- test-plan-section.ts - Converts test plan JSON to markdown with collapsible sections
- types.ts - TypeScript interfaces
- utils.ts - Helper functions (GitHub API, URL validation)
Updated build-rc-auto.yml:
- Generates test plan inline during RC build workflow
- Removed label requirement (fully automatic)
- Uploads JSON to gh-pages for automation
Deleted:
- generate-rc-test-plan.yml (no longer separate workflow)
- post-rc-build-comment.mjs (replaced by TypeScript)

What it does

Stats table (risk score, high/medium risk counts, files changed, teams signed off)
Collapsible sections for executive summary, high/medium risk scenarios
Teams sign-off status section
Excluded features (feature flags) section
JSON link at bottom for automation
Graceful fallback if test plan generation fails

Changelog

CHANGELOG entry: null

Related issues

Fixes: https://consensyssoftware.atlassian.net/browse/MCRM-69

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

#28355 (comment)

After

Should match Extension RC PR comment:
MetaMask/metamask-extension#41626 (comment)

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
I've completed the PR template to the best of my ability
I've included tests if applicable
I've documented my code using JSDoc format if applicable
I've applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Performance checks (if applicable)

I've tested on Android
- Ideally on a mid-range device; emulator is acceptable
I've tested with a power user scenario
- Use these power-user SRPs to import wallets with many accounts and tokens
I've instrumented key operations with Sentry traces for production performance metrics
- See trace() for usage and addToken for an example

For performance guidelines and tooling, see the Performance Guide.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
Changes the release CI workflow behavior and permissions (auto-triggering RC builds, generating content via external AI providers, and pushing to gh-pages), which could affect release pipeline reliability and repository writes if misconfigured.

Overview
RC builds are now fully automatic for release branches with an open PR. The build-rc-auto.yml workflow no longer gates builds on an auto-rc-builds label; instead it finds the associated PR and proceeds when one exists.

RC build announcements now optionally include an AI-generated test plan inline. The workflow replaces scripts/post-rc-build-comment.mjs with a new TypeScript scripts/build-announce/ runner that posts the build links, generates/parses release-test-plan.json via tests/tools/e2e-ai-analyzer when AI provider keys are present, formats results into collapsible markdown sections, and minimizes older RC announce comments.

Test plan hosting is consolidated into the RC build workflow. The standalone generate-rc-test-plan.yml comment-triggered workflow is removed, and the RC workflow now attempts (best-effort) to commit the generated JSON to the gh-pages branch under test-plans/ for later automation/links.

^{Reviewed by Cursor Bugbot for commit 3f0ea37. Bugbot is set up for automated code reviews on this repo. Configure here.}

github-actions · 2026-04-16T21:06:09Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8103503. Configure here.}

github-actions · 2026-04-16T21:45:30Z

🔍 Smart E2E Test Selection

Selected E2E tags: None (no tests recommended)
Selected Performance tags: None (no tests recommended)
Risk Level: low
AI Confidence: 98%

click to see 🤖 AI reasoning details

E2E Test Selection:
All 7 changed files are exclusively CI/CD pipeline automation and release scripting changes with zero impact on the application code:

.github/workflows/build-rc-auto.yml - Modified to remove the auto-rc-builds label requirement and integrate AI test plan generation into the RC build comment. Pure CI workflow change.
.github/workflows/generate-rc-test-plan.yml - Deleted workflow (functionality consolidated into build-rc-auto.yml). No app code affected.
scripts/build-announce/index.ts - New TypeScript entry point for posting RC build comments to GitHub PRs. This is a CI script, not app code.
scripts/build-announce/test-plan-section.ts - New markdown formatter for AI test plan output in PR comments. CI tooling only.
scripts/build-announce/utils.ts - New utility functions for GitHub API interactions and test plan generation. CI tooling only.
scripts/build-announce/types.ts - New TypeScript type definitions for CI scripts. No app impact.
scripts/post-rc-build-comment.mjs - Deleted (replaced by the new TypeScript module). No app impact.

None of these changes affect:

Application source code (app/)
E2E test infrastructure (tests/)
Navigation, controllers, or UI components
Any user-facing functionality

These are purely release pipeline automation changes. No E2E tests are needed to validate them.

Performance Test Selection:
No application code was changed. All changes are CI/CD pipeline scripts and GitHub workflow automation. There is no impact on app performance, rendering, data loading, or any user-facing functionality that would warrant performance testing.

View GitHub Actions results

sonarqubecloud · 2026-04-16T22:02:12Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

feat: integrate AI test plan into RC build comments

e620958

sleepytanya requested a review from a team as a code owner April 16, 2026 21:05

metamaskbotv2 Bot added the team-qa QA team label Apr 16, 2026

github-actions Bot added size-XL risk-low Low testing needed · Low bug introduction risk labels Apr 16, 2026

sleepytanya added the No QA Needed Apply this label when your PR does not need any QA effort. label Apr 16, 2026

cursor Bot reviewed Apr 16, 2026

View reviewed changes

Comment thread scripts/build-announce/utils.ts

Comment thread scripts/build-announce/index.ts

github-advanced-security AI found potential problems Apr 16, 2026

View reviewed changes

Comment thread scripts/build-announce/utils.ts Fixed

sleepytanya added 2 commits April 16, 2026 17:20

Reset testPlan to null in catch block

880a8d4

Use execFileSync to prevent command injection

6ddc906

github-actions Bot added risk-low Low testing needed · Low bug introduction risk and removed risk-low Low testing needed · Low bug introduction risk labels Apr 16, 2026

sleepytanya and others added 2 commits April 16, 2026 17:27

Show failure notice when test plan generation fails

3847871

Merge branch 'main' into release-test-plan-ga

8103503

github-actions Bot added risk-low Low testing needed · Low bug introduction risk and removed risk-low Low testing needed · Low bug introduction risk labels Apr 16, 2026

sleepytanya changed the title ~~feat: integrate AI test plan into RC build comments~~ test: integrate AI test plan into RC build comments Apr 16, 2026

sleepytanya added the release-7.74.0 Issue or pull request that will be included in release 7.74.0 label Apr 16, 2026

cursor Bot reviewed Apr 16, 2026

View reviewed changes

Comment thread scripts/build-announce/test-plan-section.ts

sleepytanya and others added 2 commits April 16, 2026 17:42

Only includes rendered high/medium risk items

1764d02

Merge branch 'main' into release-test-plan-ga

3f0ea37

github-actions Bot added risk-low Low testing needed · Low bug introduction risk and removed risk-low Low testing needed · Low bug introduction risk labels Apr 16, 2026

sleepytanya enabled auto-merge April 17, 2026 00:05

weitingsun approved these changes Apr 17, 2026

View reviewed changes

sleepytanya added this pull request to the merge queue Apr 17, 2026

Merged via the queue into main with commit 84dc598 Apr 17, 2026
68 of 69 checks passed

sleepytanya deleted the release-test-plan-ga branch April 17, 2026 21:04

github-actions Bot locked and limited conversation to collaborators Apr 17, 2026

metamaskbotv2 Bot added the release-7.75.0 Issue or pull request that will be included in release 7.75.0 label Apr 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: integrate AI test plan into RC build comments#28951

test: integrate AI test plan into RC build comments#28951
sleepytanya merged 7 commits into
mainfrom
release-test-plan-ga

sleepytanya commented Apr 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

sonarqubecloud Bot commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

sleepytanya commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Performance checks (if applicable)

Pre-merge reviewer checklist

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Apr 16, 2026

🔍 Smart E2E Test Selection

Uh oh!

sonarqubecloud Bot commented Apr 16, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sleepytanya commented Apr 16, 2026 •

edited

Loading