This issue is very much inspired by #17988. Why create a new one? #17988 was created in May 2018 and is the # 2 most liked issue with "tests" label. Unfortunately, I think it's got overloaded and just bogged down – it has 110 comments, and it's a long (yet interesting) read. I want to start from a clean slate, describe the problem again, and propose actionable solutions.

TL;DR Flutter doesn't allow for adding accessibilityIdentifier (on iOS) and resource-id (on Android) to virtual views created by Accessibility Bridge. This is problematic when doing UI testing and should be fixed.

Use case

The use case is automated end-to-end UI testing of apps built with Flutter.

There are open-source mobile testing solution that improve this situation and make end-to-end UI testing of Flutter apps feasible. I think that the 3 most popular ones are Appium, Maestro, and Patrol (I'm the author of the last).

Patrol is Flutter-first and what I'd call a greybox testing framework – it works inside the app. The benefit is that it can leverage existing flutter_test APIs. It knows about the widget/element/render trees, it knowns about pump&settle mechanism, it knows about Keys, it just "feels natural" to Flutter devs. But running inside the app has a price – e.g. if the app dies, the test dies as well (in Patrol, the test is the main() function, which loads the app – just like in integration_test). This makes it impossible to test scenarios such as removing a permission from an app – becase it kills the app.

Appium and Maestro, on the other hand, are fully black-box. They interact with the app using accessibility – they don't know anything about Flutter. This means they work well as long as the app is accessible – and most of the time it is, thanks to Flutter's great support for accessibility, enabled by Accessibility Bridge – but there's a missing piece.

The problems begin when the app grows and starts to support multiple languages, or when the visible text in the UI is no longer static (e.g. server-driven UI, A/B tests, or just changed often by business people and UI testers can't keep up with them). In such case, the solution would be to provide another identifier to a widget, in addition to its text/contentDescription. That "another identifier" is called accessibilityIdentifier on iOS and resource-id on Android (from now on I'll use the term "accessibility ID" to refer to it). The accessibility ID isn't exposed to the user, but it's visible for automatic UI testing.

Unfortunately, as of now, it's not possible to set accessibility ID from Flutter. The Semantics widget and SemanticsProperties don't have a property for it, so it can't even be forwarded to AccessibilityBridge. This makes it hard to test Flutter apps using UI testing frameworks that depend on native accessibility tree (like Appium and Maestro).

iOS

On iOS, there are a few interesting accessibility properties that can be set on a view:

• accessibility identifier
• title
• label
• value
• placeholderValue

The first one - accessibility identifier – is the most interesting for us now:

An identifier can be used to uniquely identify an element in the scripts you write using the UI Automation interfaces. Using an identifier allows you to avoid inappropriately setting or accessing an element’s accessibility label.

(source)

Currently, AccessibilityBridge on iOS doesn't set accessibilityIdentifier, because it's not even present in SemanticsProperties.

Android

On Android, the equivalent of iOS' accessibilityIdentifier is resource-id.

The situation is the same: AccessibilityBridge doesn't set resource-id on Android, because it's not present in SemanticsProperties.

Current situation

Let's consider a simple app and what its accessibility hierarchy looks like.

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Column(
          crossAxisAlignment: CrossAxisAlignment.center,
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            // The built-in Text widget automatically handles semantics
            const Text('Some text'),
            // A subtree can also be explicitly marked with semantics
            Semantics(
              label: 'Some other text',
              child: Container(
                width: 100,
                height: 100,
                color: Colors.blue,
              ),
            ),
          ],
        ),
      ),
    );
  }
}

Run that app on iOS simulator and Android emulator and retrieve the accessibility hierarchy.

iOS

On iOS, I use the idb tool to retrieve the accessibility hierarchy (and then format it nicely with jq):

$ idb ui describe-all | jq '[.[] | {accessibilityLabel: .AXLabel, accessibilityIdentifier: .AXUniqueId }]'
[
  {
    "accessibilityLabel": "Example",
    "accessibilityIdentifier": null
  },
  {
    "accessibilityLabel": "Some text",
    "accessibilityIdentifier": null
  },
  {
    "accessibilityLabel": "Some other text",
    "accessibilityIdentifier": null
  }
]

[
  {
    "AXFrame": "{{0, 0}, {390, 844}}",
    "AXUniqueId": null,
    "frame": {
      "y": 0,
      "x": 0,
      "width": 390,
      "height": 844
    },
    "role_description": "application",
    "AXLabel": "Example",
    "content_required": false,
    "type": "Application",
    "title": null,
    "help": null,
    "custom_actions": [],
    "AXValue": null,
    "enabled": true,
    "role": "AXApplication",
    "subrole": null
  },
  {
    "AXFrame": "{{15.65234375, 362}, {68.6953125, 20}}",
    "AXUniqueId": null,
    "frame": {
      "y": 362,
      "x": 15.65234375,
      "width": 68.6953125,
      "height": 20
    },
    "role_description": "text",
    "AXLabel": "Some text",
    "content_required": false,
    "type": "StaticText",
    "title": null,
    "help": null,
    "custom_actions": [],
    "AXValue": null,
    "enabled": true,
    "role": "AXStaticText",
    "subrole": null
  },
  {
    "AXFrame": "{{0, 382}, {100, 100}}",
    "AXUniqueId": null,
    "frame": {
      "y": 382,
      "x": 0,
      "width": 100,
      "height": 100
    },
    "role_description": "text",
    "AXLabel": "Some other text",
    "content_required": false,
    "type": "StaticText",
    "title": null,
    "help": null,
    "custom_actions": [],
    "AXValue": null,
    "enabled": true,
    "role": "AXStaticText",
    "subrole": null
  }
]

Android

On Android, I use the uiautomator tool:

adb shell uiautomator dump && adb pull /sdcard/window_dump.xml .

It spits out much more than on iOS, so here's the relevant part:

<node index="0" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="Some text" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[46,980][217,1032]" />

<node index="1" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="Some other text" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,1032][263,1295]" />

SemanticsProperties.label is translated into content-desc.

<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>
<hierarchy rotation="0">
    <node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
        <node index="0" text="" resource-id="" class="android.widget.LinearLayout" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
            <node index="0" text="" resource-id="android:id/content" class="android.widget.FrameLayout" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
                <node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
                    <node index="0" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
                        <node index="0" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
                            <node index="0" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
                                <node index="0" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2274]">
                                    <node index="0" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="Some text" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[46,980][217,1032]" />
                                    <node index="1" text="" resource-id="" class="android.view.View" package="com.example.example" content-desc="Some other text" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,1032][263,1295]" />
                                </node>
                            </node>
                        </node>
                    </node>
                </node>
            </node>
        </node>
        <node index="1" text="" resource-id="android:id/statusBarBackground" class="android.view.View" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,63]" />
        <node index="2" text="" resource-id="android:id/navigationBarBackground" class="android.view.View" package="com.example.example" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][0,0]" />
    </node>
</hierarchy>

Proposal

I suggest to add a new property to SemanticsProperties called accessibilityIdentifier (or maybe just identifier). This would solve #17988.

If this proposal was implemented, then the example app code from above would look like this:

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Column(
          crossAxisAlignment: CrossAxisAlignment.center,
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Semantics(
              identifier: 'some-text', // <--- NEW
              child: const Text('Some text'),
            ),
            Semantics(
              identifier: 'some-other-text', // <--- NEW
              label: 'Some other text',
              child: Container(
                width: 100,
                height: 100,
                color: Colors.blue,
              ),
            ),
          ],
        ),
      ),
    );
  }
}

iOS

This would be the result:

$ idb ui describe-all | jq '[.[] | {accessibilityLabel: .AXLabel, accessibilityIdentifier: .AXUniqueId }]'
[
  {
    "accessibilityLabel": "Example",
    "accessibilityIdentifier": null
  },
  {
    "accessibilityLabel": "Some text",
    "accessibilityIdentifier": "some-text"
  },
  {
    "accessibilityLabel": "Some other text",
    "accessibilityIdentifier": "some-other-text"
  }
]

Android

This would be the result (notice the new resource-id):

<node index="0" text="" resource-id="some-text" class="android.view.View" package="com.example.example" content-desc="Some text" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[46,980][217,1032]" />

<node index="1" text="" resource-id="some-other-text" class="android.view.View" package="com.example.example" content-desc="Some other text" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,1032][263,1295]" />

Proposal (alternative)

An alternative, interesting idea I got is to make Flutter Keys contribute to
accessibility hierarchy as accessibility identifiers.

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Column(
          crossAxisAlignment: CrossAxisAlignment.center,
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            const Text(
              'Some text',
              key: Key('some text'),
            ),
            Semantics(
              key: 'some-other-text',
              label: 'Some other text',
              child: Container(
                // or maybe the key should be here
                width: 100,
                height: 100,
                color: Colors.blue,
              ),
            ),
          ],
        ),
      ),
    );
  }
}

The results from idb and uiautomator would be the same as in the original
proposal above.

I realize that this alternative proposal might be controversial:

• wiring Keys with Flutter's accessibility system may not be something desirable
• harder to implement than simply adding a new property to SemanticsProperties and forwarding it in AccessibilityBridge
• in my understanding, Keys aren't really meant for UI testing, they're for the framework to be able to identify widgets and update them efficiently

Anyway, I'm very interested in your opinion about this approach. It doesn't have to be mutually exclusive with the original one – we could have both.

Summary

Thanks for getting this far. I think this is an important missing piece for many people who want to test their Flutter apps (just give #17988 a read for evidence). Having this would make Flutter much more friendly to automated UI testing.

I'm also willing to work on this, but first I'd like to hear if this contribution has a chance to be accepted.