Skip to content

fix(e2e-selector): handle test-only file changes in smart E2E selector#30991

Merged
cmd-ob merged 6 commits into
mainfrom
fix/smart-e2e-selector-test-file-changes
Jun 3, 2026
Merged

fix(e2e-selector): handle test-only file changes in smart E2E selector#30991
cmd-ob merged 6 commits into
mainfrom
fix/smart-e2e-selector-test-file-changes

Conversation

@cmd-ob

@cmd-ob cmd-ob commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Description

The smart E2E test selector was returning no tags when a PR changed only test files (spec files, selectors, framework utilities, flows). This meant zero E2E tests ran for test-only PRs, defeating the purpose of having a selector.

Root causes identified:

  1. findImporters only searched app/ — so when the AI investigated a changed selector/flow file it found no dependents and concluded no tests were needed
  2. The system prompt implied that changes under tests/ don't require Detox tags (a sentence scoped to wdio/ and tests/performance/ was being over-generalised)
  3. No hard rule existed for test infrastructure changes

What changed:

Two new deterministic hard rules (no AI needed):

  • test-framework-infra-change: changes to core Detox primitives (Assertions.ts, Gestures.ts, Matchers.ts, Utilities.ts, EncapsulatedElement.ts, fixtures/, config/, etc.) → run all tests
  • test-spec-tag-extraction: only spec files changed (tests/smoke/*.spec.*, tests/regression/*.spec.*) with no app code changes → extract the tag name directly from each spec's import line and run only those tags (targeted, deterministic)

AI tooling improvement:

  • findImporters now searches tests/ in addition to app/, so the AI can find which spec files import a changed selector, locator, or flow file

Prompt fix:

  • Clarified that tests/selectors/, tests/flows/, tests/locators/, tests/page-objects/ changes DO require Detox tags; only wdio/ and tests/performance/ are exempt

The HardRule interface was updated so rules return SelectTagsAnalysis | null directly, allowing targeted rules to return a partial tag list rather than always running everything.

Changelog

CHANGELOG entry: null

Related issues

Fixes:

Manual testing steps

Feature: Smart E2E selector handles test-only file changes

  Scenario: Only spec files changed
    Given a PR that only modifies tests/smoke/accounts/someTest.spec.ts
    When the smart E2E selector runs
    Then it extracts the tag (e.g. SmokeAccounts) from the spec file
    And runs only that tag without invoking the AI

  Scenario: Core framework infra changed
    Given a PR that modifies tests/framework/Assertions.ts
    When the smart E2E selector runs
    Then it triggers the test-framework-infra-change hard rule
    And runs all test tags

  Scenario: Selector file changed (targeted via AI)
    Given a PR that only modifies tests/selectors/Wallet/WalletView.selectors.ts
    When the smart E2E selector runs
    Then find_related_files returns the spec files that import the selector
    And the AI selects the appropriate tags for those specs

  Scenario: App code + spec files changed
    Given a PR that modifies both app/components/Foo.tsx and tests/smoke/foo.spec.ts
    When the smart E2E selector runs
    Then the spec-tag-extraction rule does not fire (non-test files present)
    And the AI runs and analyses both app and test changes

Screenshots/Recordings

Before

PRs changing only test files → selector returns no tags → no E2E tests run

After

  • Spec-only PRs → targeted tag extraction (deterministic)
  • Framework infra PRs → run all tests (deterministic)
  • Selector/flow PRs → AI finds importers in tests/ and selects correct tags

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
Changes CI test-selection logic; mis-tagged runs could under- or over-test PRs, but scope is limited to the e2e-ai-analyzer tool, not app runtime.

Overview
Fixes the smart E2E tag selector so test-only PRs no longer end up with zero Detox tags by adding deterministic hard rules and tightening how test changes are discovered.

Deterministic select-tags rules now run before any LLM calls (checkHardRules moved earlier in index.ts). HardRule returns a full SelectTagsAnalysis (run-all or targeted tags) instead of only a reason string. New/expanded behavior includes: core tests/framework/ changes → run all tags; spec-only smoke/regression changes → parse tag imports from specs; shared infra (flows, page-objects, selectors, locators) on test-only PRs → grep importers (up to one hop) and run only those specs’ tags.

AI/tooling: find_related_files / importers grep now includes tests/ as well as app/. Prompt guidance clarifies that Detox-relevant paths under tests/ need tags (only wdio/ and tests/performance stay exempt). Provider checks are wrapped in try/catch; default Google model is gemini-2.5-flash.

Reviewed by Cursor Bugbot for commit 1784003. Bugbot is set up for automated code reviews on this repo. Configure here.

When a PR changes only test files (selectors, spec files, framework
utilities), the AI analyzer was returning no tags — causing no E2E tests
to run. This fixes the gap with a mix of deterministic hard rules and
improved AI tooling.

Hard rules added:
- test-framework-infra-change: core Detox primitives (Assertions,
  Gestures, Matchers, Utilities, EncapsulatedElement, fixtures/,
  config/) → run all tests
- test-spec-tag-extraction: smoke/regression spec files changed with no
  app code changes → extract tags directly from the spec's import line
  and run only those (targeted, no AI needed)

Existing hard rules updated to return SelectTagsAnalysis directly so
targeted rules can also return partial results.

AI improvements:
- findImporters now searches tests/ in addition to app/, so the AI can
  find which spec files import a changed selector or flow file
- System prompt clarified: tests/selectors/, tests/flows/,
  tests/locators/, tests/page-objects/ changes DO require Detox tags;
  only wdio/ and tests/performance/ are exempt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@mm-token-exchange-service mm-token-exchange-service Bot added the team-qa QA team label Jun 3, 2026
@github-actions github-actions Bot added the size-M label Jun 3, 2026
@cmd-ob cmd-ob marked this pull request as ready for review June 3, 2026 09:41
@cmd-ob cmd-ob requested a review from a team as a code owner June 3, 2026 09:41
Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts
Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts Outdated
Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts
@github-actions github-actions Bot added the risk:high AI analysis: high risk label Jun 3, 2026
…ctors/locators

Non-deterministic AI results for shared test infra changes (e.g. wallet.flow.ts)
led to inconsistent tag selection across runs and missed regressions like #30950.

Adds a new hard rule `test-shared-infra-impact` that runs before AI and produces
the same result every time:

1. Detect changed files in tests/flows/, tests/page-objects/, tests/selectors/,
   tests/locators/ (when no app code also changed)
2. Grep for the changed filename in tests/smoke/ and tests/regression/ to find
   direct spec importers
3. Resolve one level of indirection for intermediate utility files that are not
   specs themselves
4. Extract tags from all affected spec files and return them

Uses execFileSync with an argument array (no shell interpolation) and -F
fixed-string matching — no regex injection risk.

For PR #30917: finds 150 affected specs and selects 13 tags deterministically,
which would have caught the syntax error regression introduced by the barrel
import that caused #30950.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot added risk:medium AI analysis: medium risk and removed risk:high AI analysis: high risk labels Jun 3, 2026
Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts
Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts Outdated
cmd-ob and others added 2 commits June 3, 2026 12:26
…oning output

Hard rules are deterministic and need no AI provider. Previously, all provider
availability checks (Anthropic, OpenAI, Gemini) ran before hard rules were
evaluated — wasting time and producing noisy warnings in CI.

Fix: evaluate hard rules in index.ts immediately after collecting changed files,
before any provider check. If a rule fires, output and return with no API calls.

Also tighten the reasoning string for test-shared-infra-impact: previously it
embedded all 150+ affected spec file paths, making CI logs unreadable. Now
reports a concise count instead, with the full spec list still visible in
the structured console logs above.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ni model

Two issues:
- The provider check loop had no outer try-catch, so an unexpected rejection
  from createProvider() or isAvailable() could crash the entire run. Wrapped
  each provider check in try-catch so one broken provider is skipped cleanly.
- gemini-2.0-flash returns 404 (model removed). Updated to gemini-2.5-flash.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.86%. Comparing base (dd6e274) to head (b8e5ddb).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #30991   +/-   ##
=======================================
  Coverage   82.86%   82.86%           
=======================================
  Files        5570     5570           
  Lines      143557   143557           
  Branches    33299    33299           
=======================================
+ Hits       118960   118961    +1     
  Misses      16629    16629           
+ Partials     7968     7967    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- Bug 1: extend extractTagsFromSpecFile regex to match tags.js imports
  with explicit .js extension (\/tags(?:\.js)?['"])
- Bug 3: extend findSpecFilesImporting step 1 to also grep in
  tests/page-objects/, tests/flows/, tests/selectors/, tests/locators/
  so selector → page-object → spec chains are resolved correctly
- Bug 4: test-shared-infra-impact returns null (not empty result) when
  no tags found, allowing test-spec-tag-extraction to run next
- Bug 5: test-shared-infra-impact now unions tags from directly-changed
  spec files alongside transitively-affected spec files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot added risk:low AI analysis: low risk and removed risk:medium AI analysis: medium risk labels Jun 3, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ad2a60d. Configure here.

Comment thread tests/tools/e2e-ai-analyzer/modes/select-tags/handlers.ts
…alse positives

Grepping for a bare stem (e.g. 'index', 'utils') matches anywhere in file
content — comments, variable names, strings — pulling in unrelated spec files.
Prefix the search pattern with '/' so it only matches import paths like
from './TokenSelectors' or from '../../selectors/TokenSelectors'.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

🔍 Smart E2E Test Selection

  • Selected E2E tags: None (no tests recommended)
  • Selected Performance tags: None (no tests recommended)
  • Risk Level: low
  • AI Confidence: 97%
click to see 🤖 AI reasoning details

E2E Test Selection:
All 7 changed files are exclusively within tests/tools/e2e-ai-analyzer/ — this is the AI-powered test tag selection tool itself (meta-tooling), not the MetaMask Mobile app code or any E2E test specs/infrastructure.

Key changes:

  1. config.ts: Gemini model bump from gemini-2.0-flashgemini-2.5-flash — no app impact.
  2. types/index.ts: HardRule.check() return type changed from string | null to SelectTagsAnalysis | null — internal type refactor of the analyzer tool.
  3. test-infrastructure-paths.ts (new file): New module classifying test infra file categories — only used by the analyzer tool.
  4. handlers.ts: New hard rules (test-framework-infra-change, test-shared-infra-impact, test-spec-tag-extraction), helper functions for spec file discovery and tag extraction — all internal to the analyzer tool.
  5. prompt.ts: One line of guidance text added to the AI system prompt — no runtime test impact.
  6. index.ts: Hard rule check moved before provider availability check (optimization), try/catch added around provider checks — internal tool flow improvement.
  7. related-files.ts: grep extended to search tests/ in addition to app/ — internal tool improvement.

None of these changes touch: app source code, E2E spec files, test framework utilities (Assertions, Gestures, Matchers, fixtures), page objects, flows, selectors, controllers, Engine, navigation, or any component that could affect test outcomes. The changes are purely to the meta-tooling that decides which tests to run. No E2E tests need to run to validate these changes, and there is zero performance impact on the app.

Performance Test Selection:
All changes are within the e2e-ai-analyzer tool (meta-tooling). No app code, UI components, controllers, state management, or critical user flows were modified. Zero performance impact on the MetaMask Mobile app.

View GitHub Actions results

@cmd-ob cmd-ob added this pull request to the merge queue Jun 3, 2026
Merged via the queue into main with commit 19e2565 Jun 3, 2026
159 checks passed
@cmd-ob cmd-ob deleted the fix/smart-e2e-selector-test-file-changes branch June 3, 2026 13:03
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 3, 2026
@metamaskbotv2 metamaskbotv2 Bot added the release-7.81.0 Issue or pull request that will be included in release 7.81.0 label Jun 3, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

release-7.81.0 Issue or pull request that will be included in release 7.81.0 risk:low AI analysis: low risk size-M team-qa QA team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants