Skip to content

Add android_hardware_smoke_test integration tests#187130

Merged
andywolff merged 65 commits into
flutter:masterfrom
andywolff:apktest
Jun 3, 2026
Merged

Add android_hardware_smoke_test integration tests#187130
andywolff merged 65 commits into
flutter:masterfrom
andywolff:apktest

Conversation

@andywolff

@andywolff andywolff commented May 26, 2026

Copy link
Copy Markdown
Contributor

Create a new suite of integration tests which can run either in CI or directly on an android device as APKs. See the readme for more details on how it works.

This begins to address #182123

We'll expand coverage in future PRs after we verify that this runs well in CI.

Pre-launch Checklist

@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label May 26, 2026
@github-actions github-actions Bot removed the CICD Run CI/CD label May 26, 2026
@andywolff andywolff added the CICD Run CI/CD label May 26, 2026
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the android_hardware_smoke_test integration test suite, which supports both standalone on-device testing for OEMs and host-driven CI testing. It registers Vulkan and OpenGLES test shards, sets up custom Gradle tasks to embed test screenshots into HTML reports, and implements the necessary Dart and Android native harnesses. The review feedback identifies a critical regex mismatch that silently breaks backend selection, warns of binary corruption on Windows hosts when using adb shell instead of adb exec-out, recommends robust error handling and null safety in the golden comparison logic to prevent test hangs, and suggests minor style guide and dependency pinning improvements.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
Comment thread dev/integration_tests/android_hardware_smoke_test/android/app/build.gradle.kts Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart
Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
Comment thread dev/integration_tests/android_hardware_smoke_test/README.md Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart
Comment thread dev/integration_tests/android_hardware_smoke_test/android/app/build.gradle.kts Outdated
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot removed the CICD Run CI/CD label May 27, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new integration and compatibility smoke test suite, android_hardware_smoke_test, designed to verify visual rendering correctness on Android hardware using both an on-device instrumented mode and a host-driven driver mode. The feedback suggests refining the regular expression used to match the ImpellerBackend metadata tag to prevent greedy matching, using JUnit 4 assertions consistently in FlutterActivityTest.java, increasing test timeouts to avoid CI flakiness, and checking the exit code of the adb exec-out process in the Gradle task to handle potential transfer failures gracefully.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart Outdated
@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label May 27, 2026
@github-actions github-actions Bot added framework flutter/packages/flutter repository. See also f: labels. a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) CICD Run CI/CD and removed CICD Run CI/CD labels May 27, 2026
@github-actions github-actions Bot added CICD Run CI/CD and removed CICD Run CI/CD framework flutter/packages/flutter repository. See also f: labels. a: accessibility Accessibility, e.g. VoiceOver or TalkBack. (aka a11y) labels May 27, 2026
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the android_hardware_smoke_test integration test suite, designed to verify visual rendering correctness on Android hardware using both a self-contained instrumented mode and a host-driven driver mode. The changes include the test suite setup, native Android activity configurations, and a custom Gradle task to embed result screenshots into the HTML reports. Feedback is provided on improving the robustness of the regular expression used to match the Impeller backend, resolving potential thread/resource leaks in the Java test executor, adhering to Google Java Style Guide spacing rules for binary operators, and improving error handling when loading golden assets or casting render objects.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart Outdated
@github-actions github-actions Bot removed the CICD Run CI/CD label May 28, 2026
@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label May 28, 2026
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new integration and compatibility smoke test suite, android_hardware_smoke_test, to verify visual rendering correctness on Android hardware using a dual-mode architecture (instrumented and driver modes). Key feedback includes making a regular expression in the suite runner non-greedy, refactoring golden image comparison helpers to return a Future instead of passing a Completer to avoid potential double-completion errors, explicitly specifying offset and length when converting ByteData to Uint8List to prevent reading incorrect buffer data, and adding a null check for activity.messageChannel in the native test harness to prevent a potential NullPointerException.

Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart Outdated
@github-actions github-actions Bot removed the CICD Run CI/CD label May 28, 2026
@andywolff

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the android_hardware_smoke_test integration test suite to verify visual rendering correctness on Android hardware. Feedback on the changes includes addressing a critical Dart syntax error where Swift-style implicit member syntax is used instead of ColorScheme.fromSeed. Additionally, suggestions are provided to make the regular expression in the test runner more robust, simplify redundant try-catch blocks in goldens.dart, and improve the error handling in the Gradle build script when querying sandbox files via ADB.

Comment thread dev/integration_tests/android_hardware_smoke_test/lib/main.dart
Comment thread dev/bots/suite_runners/run_android_hardware_smoke_tests.dart
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart Outdated
Comment thread dev/integration_tests/android_hardware_smoke_test/lib/goldens.dart Outdated
@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label Jun 2, 2026
@andywolff andywolff requested a review from gaaclarke June 2, 2026 16:41
@andywolff

Copy link
Copy Markdown
Contributor Author

Verified CI using led tool to trigger staging runs of the new shards. They behaved as expected, failing due to lack of goldens in skia gold. We'll approve the new goldens after merging and the new shards run in postsubmit

@gaaclarke gaaclarke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! thanks andy

@andywolff andywolff added this pull request to the merge queue Jun 2, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 2, 2026
@github-actions github-actions Bot removed the CICD Run CI/CD label Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants