Rather than [run] here, I think we probably want a link to either API docs or a wiki page about this.

Also, now that this lives in the Flutter repo, we may want the flutter drive command to handle this - /cc @jonahwilliams for thoughts on that.

I'm not sure the context, but we could accept more command line args for the drive script if that is what you mean?

@jiahaog would like to extend some functionality to make it easier to run integration tests - in particular, to wrap the test script with this kind of logic, and to help find all integration tests and run them (the way it works with other _test.dart files for regular unit tests).

@jonahwilliams There's more detailed context in #66264.

The short summary is that flutter drive is a pretty low level CLI that is not too friendly for users when they need to specify different tests / driver scripts. In addition, there's a bit of boilerplate that can we can hide from users in some sort of executable that we provide. With this PR, users will need to wrap their tests with run

Before

import 'package:integration_test/integration_test.dart';

void main() {
  testWidgets("failing test example", (WidgetTester tester) async { ... });
}

After

import 'package:integration_test/integration_test.dart';

void main() => run(_testMain);

void _testMain() {
  testWidgets("failing test example", (WidgetTester tester) async { ... });
}

It will be great if we could hide the need to call run from users, in a "bootstrap" sort of functionality in the executable – I believe package:flutter_test and package:test do something similar in the executables that are provided.

Ahh okay. No I think putting this in integration_test is the right way to go. The tool API should remain as is - and generally speaking we should avoid increasing that surface area because it will be difficult/impossible to change

I'm not familiar with the flutter release process, will having a separate tool for integration tests in packages/integration_test/bin/ be the correct approach here? (Users will be invoking it with something like flutter pub run integration_test)

nit: formatting

Is there any way we could avoid switching on types here? That can become brittle. Could we instead have the results be of such a type that it has some property saying whether it's an error?

Maybe rather than a separate Failure type, we have a .failure ctor?

Assuming you mean something like this:

enum ResultStatus {
  pass, fail
}

class Result {
  final ResultStatus status;

  Result.failure(...);

  Result.success(...);
}

I'm all for this, but the inconvenient bit is that Failure today is already part of the public type, and we might not want to replace Failure with the Result above.

Dart doesn't have type unions, but another option would be to have some sort of superclass type:

class Result {}

// make the old failure extend [Result]
class Failure extends Result {}

class Success extends Result {}

And then store them in the binding as Map<String, Result>, or even List<Result> (since the key for the test name can be contained within the class). What do you think?

I think having the supertype is fine. We could then just add some bool getter on it about whether the result describes a failure, and make Failure implement it as true.

Added the supertype. I think the bool getter is a bit unnecessary and switching on types should be alright – now that the parent type is Result checking for Failure should be safer

Also, this seems concerning that it will only have the first error. It seems like it would be better to capture them all.

Resolved the formatting issue.

Also, this seems concerning that it will only have the first error. It seems like it would be better to capture them all.

I took this snippet from the Google3 test reporter. Will do a bit of research to find out if this is necessary and follow up :)

@natebosch may know.

I think in any system which expects only a single reason for a test case to fail it is sensible to choose the first error to report. The Dart test runner allows a single test to have multiple errors, but if we are reporting into a system that only allows a single reason to fail we have pick between trying to do a strange concatenation or only reporting the first error.

I think we should probably support whatever errors we can access internally, and then we can either bundle them up or not when we report to JUint/XCTest/etc.

Made this collect the list of errors. I did not wire the errors into the platform side as it's a bit out of scope.