fix(e2e-selector): handle test-only file changes in smart E2E selector#30991
Conversation
When a PR changes only test files (selectors, spec files, framework utilities), the AI analyzer was returning no tags — causing no E2E tests to run. This fixes the gap with a mix of deterministic hard rules and improved AI tooling. Hard rules added: - test-framework-infra-change: core Detox primitives (Assertions, Gestures, Matchers, Utilities, EncapsulatedElement, fixtures/, config/) → run all tests - test-spec-tag-extraction: smoke/regression spec files changed with no app code changes → extract tags directly from the spec's import line and run only those (targeted, no AI needed) Existing hard rules updated to return SelectTagsAnalysis directly so targeted rules can also return partial results. AI improvements: - findImporters now searches tests/ in addition to app/, so the AI can find which spec files import a changed selector or flow file - System prompt clarified: tests/selectors/, tests/flows/, tests/locators/, tests/page-objects/ changes DO require Detox tags; only wdio/ and tests/performance/ are exempt Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
…ctors/locators Non-deterministic AI results for shared test infra changes (e.g. wallet.flow.ts) led to inconsistent tag selection across runs and missed regressions like #30950. Adds a new hard rule `test-shared-infra-impact` that runs before AI and produces the same result every time: 1. Detect changed files in tests/flows/, tests/page-objects/, tests/selectors/, tests/locators/ (when no app code also changed) 2. Grep for the changed filename in tests/smoke/ and tests/regression/ to find direct spec importers 3. Resolve one level of indirection for intermediate utility files that are not specs themselves 4. Extract tags from all affected spec files and return them Uses execFileSync with an argument array (no shell interpolation) and -F fixed-string matching — no regex injection risk. For PR #30917: finds 150 affected specs and selects 13 tags deterministically, which would have caught the syntax error regression introduced by the barrel import that caused #30950. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oning output Hard rules are deterministic and need no AI provider. Previously, all provider availability checks (Anthropic, OpenAI, Gemini) ran before hard rules were evaluated — wasting time and producing noisy warnings in CI. Fix: evaluate hard rules in index.ts immediately after collecting changed files, before any provider check. If a rule fires, output and return with no API calls. Also tighten the reasoning string for test-shared-infra-impact: previously it embedded all 150+ affected spec file paths, making CI logs unreadable. Now reports a concise count instead, with the full spec list still visible in the structured console logs above. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ni model Two issues: - The provider check loop had no outer try-catch, so an unexpected rejection from createProvider() or isAvailable() could crash the entire run. Wrapped each provider check in try-catch so one broken provider is skipped cleanly. - gemini-2.0-flash returns 404 (model removed). Updated to gemini-2.5-flash. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #30991 +/- ##
=======================================
Coverage 82.86% 82.86%
=======================================
Files 5570 5570
Lines 143557 143557
Branches 33299 33299
=======================================
+ Hits 118960 118961 +1
Misses 16629 16629
+ Partials 7968 7967 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Bug 1: extend extractTagsFromSpecFile regex to match tags.js imports with explicit .js extension (\/tags(?:\.js)?['"]) - Bug 3: extend findSpecFilesImporting step 1 to also grep in tests/page-objects/, tests/flows/, tests/selectors/, tests/locators/ so selector → page-object → spec chains are resolved correctly - Bug 4: test-shared-infra-impact returns null (not empty result) when no tags found, allowing test-spec-tag-extraction to run next - Bug 5: test-shared-infra-impact now unions tags from directly-changed spec files alongside transitively-affected spec files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ad2a60d. Configure here.
…alse positives Grepping for a bare stem (e.g. 'index', 'utils') matches anywhere in file content — comments, variable names, strings — pulling in unrelated spec files. Prefix the search pattern with '/' so it only matches import paths like from './TokenSelectors' or from '../../selectors/TokenSelectors'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🔍 Smart E2E Test Selection
click to see 🤖 AI reasoning detailsE2E Test Selection: Key changes:
None of these changes touch: app source code, E2E spec files, test framework utilities (Assertions, Gestures, Matchers, fixtures), page objects, flows, selectors, controllers, Engine, navigation, or any component that could affect test outcomes. The changes are purely to the meta-tooling that decides which tests to run. No E2E tests need to run to validate these changes, and there is zero performance impact on the app. Performance Test Selection: |

Description
The smart E2E test selector was returning no tags when a PR changed only test files (spec files, selectors, framework utilities, flows). This meant zero E2E tests ran for test-only PRs, defeating the purpose of having a selector.
Root causes identified:
findImportersonly searchedapp/— so when the AI investigated a changed selector/flow file it found no dependents and concluded no tests were neededtests/don't require Detox tags (a sentence scoped towdio/andtests/performance/was being over-generalised)What changed:
Two new deterministic hard rules (no AI needed):
test-framework-infra-change: changes to core Detox primitives (Assertions.ts,Gestures.ts,Matchers.ts,Utilities.ts,EncapsulatedElement.ts,fixtures/,config/, etc.) → run all teststest-spec-tag-extraction: only spec files changed (tests/smoke/*.spec.*,tests/regression/*.spec.*) with no app code changes → extract the tag name directly from each spec's import line and run only those tags (targeted, deterministic)AI tooling improvement:
findImportersnow searchestests/in addition toapp/, so the AI can find which spec files import a changed selector, locator, or flow filePrompt fix:
tests/selectors/,tests/flows/,tests/locators/,tests/page-objects/changes DO require Detox tags; onlywdio/andtests/performance/are exemptThe
HardRuleinterface was updated so rules returnSelectTagsAnalysis | nulldirectly, allowing targeted rules to return a partial tag list rather than always running everything.Changelog
CHANGELOG entry: null
Related issues
Fixes:
Manual testing steps
Screenshots/Recordings
Before
PRs changing only test files → selector returns no tags → no E2E tests run
After
Pre-merge author checklist
Pre-merge reviewer checklist
Note
Medium Risk
Changes CI test-selection logic; mis-tagged runs could under- or over-test PRs, but scope is limited to the e2e-ai-analyzer tool, not app runtime.
Overview
Fixes the smart E2E tag selector so test-only PRs no longer end up with zero Detox tags by adding deterministic hard rules and tightening how test changes are discovered.
Deterministic select-tags rules now run before any LLM calls (
checkHardRulesmoved earlier inindex.ts).HardRulereturns a fullSelectTagsAnalysis(run-all or targeted tags) instead of only a reason string. New/expanded behavior includes: coretests/framework/changes → run all tags; spec-only smoke/regression changes → parse tag imports from specs; shared infra (flows, page-objects, selectors, locators) on test-only PRs → grep importers (up to one hop) and run only those specs’ tags.AI/tooling:
find_related_files/ importers grep now includestests/as well asapp/. Prompt guidance clarifies that Detox-relevant paths undertests/need tags (onlywdio/andtests/performancestay exempt). Provider checks are wrapped in try/catch; default Google model isgemini-2.5-flash.Reviewed by Cursor Bugbot for commit 1784003. Bugbot is set up for automated code reviews on this repo. Configure here.