Skip to content

ci: add QA stats workflow to collect tests metrics#26559

Merged
jvbriones merged 12 commits into
mainfrom
claude/stupefied-bhabha
Mar 2, 2026
Merged

ci: add QA stats workflow to collect tests metrics#26559
jvbriones merged 12 commits into
mainfrom
claude/stupefied-bhabha

Conversation

@jvbriones

@jvbriones jvbriones commented Feb 25, 2026

Copy link
Copy Markdown
Contributor

Description

Introduces an automated pipeline that collects test metrics after every successful push to main and writes them to a qa-stats.json artifact.

  • New qa-stats.yml workflow — triggers after CI succeeds on main (excludes PRs, forks, failed runs)
  • New collect-qa-stats.mjs — extensible script that fetches CI artifacts via GitHub API (paginated) and collects component_view_tests_count and unit_tests_count (passed + failed, excludes skipped/todo). Collector failures are isolated and don't crash the run
  • ci.yml — shard jobs now extract a minimal {"count": N} via jq instead of uploading full Jest JSON; removed redundant merge-cv-test-coverage job (folded into merge-unit-and-component-view-tests), saving one yarn install per CI run; renamed coverage artifact to lcov.info

Changelog

CHANGELOG entry: null

Related issues

Fixes:

Manual testing steps

Feature: QA Stats collection on main

  Scenario: CI completes successfully on main
    Given a push to the main branch triggers the ci workflow
    And all cv-test shards complete successfully

    When the merge-cv-test-coverage job runs
    Then a cv-test-stats artifact is uploaded containing { "component_view_test_number": N }

    When the QA Stats workflow is triggered via workflow_run
    Then a qa-stats artifact is uploaded containing { "component_view_test_number": N }

Screenshots/Recordings

Before

N/A — new workflow, no prior state.

After

N/A — CI-only change; artifacts visible in GitHub Actions run summary after merging to main.

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
CI/workflow logic is reworked to produce and aggregate per-shard artifacts and to add a new workflow_run pipeline; failures could break coverage aggregation, artifact naming expectations, or downstream CI consumers.

Overview
After successful main CI runs, a new QA Stats workflow_run pipeline downloads CI artifacts and publishes qa-stats.json containing unit and component-view test counts.

ci.yml now emits per-shard test result JSON, extracts passed+failed test counts into small count.json files, aggregates totals in merge-unit-and-component-view-tests, and uploads new artifacts (cv-test-stats, unit-test-stats, lcov.info, plus CV HTML coverage); it also removes the separate CV-coverage merge job and renames coverage artifacts accordingly.

Adds .github/scripts/collect-qa-stats.mjs to fetch artifacts via the GitHub API (paginated, manual redirect handling) and tolerate missing metrics by skipping failed collectors.

Written by Cursor Bugbot for commit bfdaaa6. This will update automatically on new commits. Configure here.

Adds a new qa-stats.yml workflow triggered via workflow_run after the ci
workflow completes on main. Modifies ci.yml to output Jest test results JSON
per shard and aggregate them in the merge-cv-test-coverage job into a
cv-test-stats artifact. The collect-qa-stats.mjs script reads the pre-aggregated
stats and writes qa-stats.json for consumption by downstream workflows.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jvbriones jvbriones requested a review from a team as a code owner February 25, 2026 12:55
@github-actions

Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@metamaskbot metamaskbot added the team-qa QA team label Feb 25, 2026
Comment thread .github/workflows/qa-stats.yml Outdated
@jvbriones jvbriones added the no-changelog no-changelog Indicates no external facing user changes, therefore no changelog documentation needed label Mar 2, 2026
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml
Comment thread .github/scripts/collect-qa-stats.mjs
Comment thread .github/workflows/ci.yml
echo "No changes detected"
fi

js-bundle-size-check:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just location change, for consistency with jobs that run first in the workflow

@jvbriones jvbriones changed the title ci: add QA stats workflow to collect CV test scenario counts on main ci: add QA stats workflow to collect tests metrics Mar 2, 2026
Comment thread .github/scripts/collect-qa-stats.mjs Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread .github/scripts/collect-qa-stats.mjs
tommasini
tommasini previously approved these changes Mar 2, 2026
@jvbriones jvbriones enabled auto-merge March 2, 2026 16:56
Cal-L
Cal-L previously approved these changes Mar 2, 2026

@Cal-L Cal-L left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jvbriones jvbriones dismissed stale reviews from Cal-L and tommasini via 9e48f0a March 2, 2026 17:16
@jvbriones jvbriones added the skip-e2e skip E2E test jobs label Mar 2, 2026
@github-actions

github-actions Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

🔍 Smart E2E Test Selection

  • Selected E2E tags: None (no tests recommended)
  • Selected Performance tags: None (no tests recommended)
  • Risk Level: low
  • AI Confidence: 95%
click to see 🤖 AI reasoning details

E2E Test Selection:
The changes in this PR are purely CI/infrastructure related and do not affect any application code or E2E test execution:

  1. ci.yml changes:

    • Formatting/whitespace improvements (blank lines between jobs)
    • Job reorganization (moving jobs to different positions in the file)
    • Renaming cv-test to component-view-tests (cosmetic)
    • Adding test count collection logic (post-test processing)
    • Artifact naming changes (coverage-* to coverage-unit-* and coverage-cv-*)
    • Merging merge-cv-test-coverage job into merge-unit-and-component-view-tests
    • Adding CV test HTML coverage report generation
  2. qa-stats.yml: New workflow that runs AFTER CI completes on main branch to collect QA statistics. This is a downstream workflow that doesn't affect test execution.

  3. collect-qa-stats.mjs: New script to download and aggregate test count artifacts from CI runs. This is purely for metrics collection.

None of these changes:

  • Modify application source code
  • Change E2E test framework or fixtures
  • Affect how tests are executed
  • Impact any user-facing functionality

The CI pipeline itself will validate these changes by running successfully. No E2E tests are needed to verify CI infrastructure changes that don't touch the application or test execution logic.

Performance Test Selection:
These changes are purely CI/infrastructure related (workflow files and a metrics collection script). They do not affect any application code, UI components, data loading, state management, or any other aspect that could impact app performance. No performance tests are needed.

View GitHub Actions results

@jvbriones jvbriones added this pull request to the merge queue Mar 2, 2026
Merged via the queue into main with commit fc33bd8 Mar 2, 2026
59 checks passed
@jvbriones jvbriones deleted the claude/stupefied-bhabha branch March 2, 2026 17:59
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 2, 2026
@metamaskbot metamaskbot added the release-7.69.0 Issue or pull request that will be included in release 7.69.0 label Mar 2, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

no-changelog no-changelog Indicates no external facing user changes, therefore no changelog documentation needed release-7.69.0 Issue or pull request that will be included in release 7.69.0 size-M skip-e2e skip E2E test jobs team-qa QA team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants