test: add coverage report for feature flags in e2e by pnarayanaswamy · Pull Request #27970 · MetaMask/metamask-mobile

pnarayanaswamy · 2026-03-26T12:24:13Z

Description

Changelog

CHANGELOG entry:

Related issues

Fixes:

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

After

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
I've completed the PR template to the best of my ability
I've included tests if applicable
I've documented my code using JSDoc format if applicable
I've applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Low Risk
Low risk: changes are confined to QA metrics collection and a scheduled GitHub Actions workflow, but it adds a yarn install + ts-node execution step that could fail if dependencies or paths drift.

Overview
Adds feature-flag E2E coverage reporting to QA stats. The QA stats collector now runs a new ts-node-based scanner (tests/feature-flags/feature-flag-coverage-report.ts) that parses tests/smoke/tests/regression for references to FEATURE_FLAG_REGISTRY, classifies per-flag coverage (full/partial/default-only), writes tests/artifacts/feature-flag-coverage-report.json, and emits aggregated metrics under a new feature_flags namespace in qa-stats.json.

The qa-stats.yml scheduled workflow is updated to cache/install Yarn dependencies (required for ts-node) and to upload the feature-flag coverage JSON as a separate artifact.

^{Written by Cursor Bugbot for commit 27ecb4f. This will update automatically on new commits. Configure here.}

github-actions · 2026-03-26T12:24:24Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

jvbriones · 2026-03-30T09:21:36Z

 scripts/reports/

+# Feature flag coverage report (generated)
+tests/feature-flags/feature-flag-coverage-report.json


can we use /tmp folder instead or another?

github-actions · 2026-03-30T11:41:51Z

🔍 Smart E2E Test Selection

Selected E2E tags: None (no tests recommended)
Selected Performance tags: None (no tests recommended)
Risk Level: low
AI Confidence: 97%

click to see 🤖 AI reasoning details

E2E Test Selection:
All 4 changed files are purely CI/tooling infrastructure with zero impact on app code or E2E test logic:

.github/workflows/qa-stats.yml: Adds yarn install --immutable step and a new artifact upload for the feature flag coverage report. This is a CI workflow improvement that doesn't affect how E2E tests run.
package.json: Adds a single new npm script test:e2e:feature-flag:coverage. No dependency changes, no app code changes.
tests/feature-flags/feature-flag-coverage-report.ts: A brand-new reporting/analysis tool that scans E2E test files for feature flag references and generates a JSON coverage report. It is a read-only analysis script — it does not modify tests, fixtures, or app code.
.github/scripts/collect-qa-stats.mjs: Adds a collectFeatureFlagCoverage() function to the QA stats collector, integrating the new report into the existing stats pipeline.

None of these changes touch: app source code, E2E test files, test fixtures, page objects, test helpers, controllers, navigation, or any shared component. There is no risk of breaking existing E2E tests. No E2E test tags are needed.

Performance Test Selection:
No app source code was modified. All changes are CI workflow and tooling scripts. There is no impact on UI rendering, data loading, state management, or any user-facing flow that would affect performance metrics.

View GitHub Actions results

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

cursor · 2026-03-30T11:46:19Z

+    console.log('  PARTIAL COVERAGE (only one state tested)');
+    console.log(`  ${'-'.repeat(40)}`);
+    for (const entry of partialFlags) {
+      const missing = entry.testedStates.true ? 'false' : 'true';


Misleading "missing state" message for partial coverage flags

Low Severity

When a flag has a non-boolean productionDefault (e.g., arrays like additionalNetworksBlacklist: []) and is referenced in test files without extractable truthy/falsy values, determineCoverage classifies it as 'partial' even though testedStates is {true: false, false: false} — neither state is actually tested. Then printReport computes const missing = entry.testedStates.true ? 'false' : 'true', which assumes one state IS tested, producing a misleading "missing: true state" message when both states are untested. This also inflates coveragePercentage since these flags are counted in partialCount.

Additional Locations (1)

tests/feature-flags/feature-flag-coverage-report.ts#L232-L244

sonarqubecloud · 2026-03-30T12:05:31Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

test: add coverage report for feature flags in e2e

0603351

metamaskbot added the team-qa QA team label Mar 26, 2026

github-actions Bot added size-L risk-low Low testing needed · Low bug introduction risk labels Mar 26, 2026

cursor Bot reviewed Mar 26, 2026

View reviewed changes

Comment thread .github/scripts/collect-qa-stats.mjs Outdated

test: update flow

bb9018b

pnarayanaswamy requested a review from a team as a code owner March 26, 2026 12:34

github-actions Bot added risk-low Low testing needed · Low bug introduction risk and removed risk-low Low testing needed · Low bug introduction risk labels Mar 26, 2026

pnarayanaswamy added the no-changelog no-changelog Indicates no external facing user changes, therefore no changelog documentation needed label Mar 26, 2026

jvbriones reviewed Mar 30, 2026

View reviewed changes

jvbriones previously approved these changes Mar 30, 2026

View reviewed changes

fix: address review

27ecb4f

pnarayanaswamy dismissed jvbriones’s stale review via 27ecb4f March 30, 2026 11:40

github-actions Bot added risk-low Low testing needed · Low bug introduction risk and removed risk-low Low testing needed · Low bug introduction risk labels Mar 30, 2026

cursor Bot reviewed Mar 30, 2026

View reviewed changes

jvbriones approved these changes Mar 30, 2026

View reviewed changes

jvbriones enabled auto-merge March 30, 2026 12:52

asalsys approved these changes Mar 30, 2026

View reviewed changes

jvbriones added this pull request to the merge queue Mar 30, 2026

Merged via the queue into main with commit 5800a40 Mar 30, 2026
63 checks passed

jvbriones deleted the add-ff-coverage branch March 30, 2026 14:26

github-actions Bot locked and limited conversation to collaborators Mar 30, 2026

metamaskbot added the release-7.73.0 Issue or pull request that will be included in release 7.73.0 label Mar 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: add coverage report for feature flags in e2e#27970

test: add coverage report for feature flags in e2e#27970
jvbriones merged 3 commits into
mainfrom
add-ff-coverage

pnarayanaswamy commented Mar 26, 2026 •

edited by cursor Bot

Loading

Uh oh!

github-actions Bot commented Mar 26, 2026

Uh oh!

Uh oh!

jvbriones Mar 30, 2026

Uh oh!

pnarayanaswamy Mar 30, 2026

Uh oh!

github-actions Bot commented Mar 30, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Mar 30, 2026

Uh oh!

sonarqubecloud Bot commented Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

pnarayanaswamy commented Mar 26, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

Uh oh!

github-actions Bot commented Mar 26, 2026

Uh oh!

Uh oh!

jvbriones Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

pnarayanaswamy Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 30, 2026

🔍 Smart E2E Test Selection

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Mar 30, 2026

Choose a reason for hiding this comment

Misleading "missing state" message for partial coverage flags

Uh oh!

sonarqubecloud Bot commented Mar 30, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pnarayanaswamy commented Mar 26, 2026 •

edited by cursor Bot

Loading