[integration_test] Add a `run` method for proper reporting of test results #3179

jiahaog · 2020-10-21T13:12:14Z

Description

This integrates with directRunTests from package:test_core, and allows users to use test() to wrap their tests, instead of only supporting testWidgets() and failing silently otherwise. In addition, exceptions thrown in other lifecycle blocks such as setUp will be caught and reported.

This is a non-breaking change, so the existing reporting mechanism will continue to work. A global variable is used to ensure that using run will disable the existing reporting mechanism.

Example usage

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

void main() => run(_testMain);

void _testMain() {
  test('A test', () {
    expect(true, true);
  });
}

But we will want to make it easier for users to use this API. A CLI will be added as a follow up PR to wrap this method as part of flutter/flutter#66264, so users can write the following instead, and the CLI will take care of wrapping the users' script with run appropriately.

import 'package:flutter_test/flutter_test.dart';

void main() {
  test('A test', () {
    expect(true, true);
  });
}

Related Issues

Fixes flutter/flutter#50816.

Breaking Change

Does your PR require plugin users to manually update their apps to accommodate your change?

No, this is not a breaking change.

…sults This integrates with `directRunTests` from `package:test_core`, and allows users to use `test()` to wrap their tests, instead of only supporting `testWidgets()` and failing silently otherwise. In addition, exceptions thrown in other lifecycle blocks such as `setUp` will be caught and reported. This is a non-breaking change, so the existing reporting mechanism will continue to work. A global variable is used to ensure that using `run` will disable the existing reporting mechanism. Fixes #50816.

jiahaog

Sending this draft for review first for preliminary feedback – this is still pending a release from package:test_core so that we can use directRunTests.

I also want to restructure the tests a bit, and do a bit more testing locally to ensure that nothing is broken

packages/integration_test/example/pubspec.yaml

packages/integration_test/pubspec.yaml

packages/integration_test/lib/integration_test.dart

dnfield · 2020-10-21T15:55:39Z

packages/integration_test/lib/integration_test.dart

+      },
+    );
+  } on MissingPluginException {
+    print('Warning: integration_test test plugin was not detected.');


I think this should be treated as a more serious failure, and probably make us exit with some non-zero exit code.

The risk is that if this isn't set up right in the application, it will print what looks like a warning in a sea of log output and exit with zero when tests may have failed.

We don't want the application to exit when it is invoked using flutter driver, as the native plugin is not used. The platform side plugin is only used when running with FTL or through Xcode. The responsibility of failing when the infrastructure is not set up properly (i.e. the native plugin is not installed) should probably be up to the host side invoker though, and probably not here.

packages/integration_test/lib/integration_test.dart

dnfield · 2020-10-21T15:59:55Z

packages/integration_test/lib/src/reporter.dart

+import 'package:test_core/src/runner/engine.dart';
+import 'package:test_core/src/runner/reporter.dart';


This will give warnings right?

We should probably pin the dependency for this to make sure we don't inadventently get broken by a patch release that changes internals.

I'm also surprised we don't need analysis ignores here, like // ignore: implementation_import or something. We should enable that lint and add the ignore here.

Yeah the lint isn't set up for this. This is a bit out of scope of this PR though – I have started #3187 to see if we can turn on the lint. There are some existing violation for this lint in the repo, will follow up tomorrow.

dnfield · 2020-10-21T16:01:13Z

packages/integration_test/lib/src/reporter.dart

+        liveTest.test.name: liveTest.state.result.name == success
+            ? success
+            : Failure(
+                liveTest.test.name, liveTest.errors.first.stackTrace.toString(),


Is the first stack trace the right one?

Is the first error the right one?

Is there some way we could add all stack traces and errors to the reporting?

This was taken from the way other Dart tests are integrated with the Google3 infrastructure. I'm not sure if we should do anything else, @natebosch would you know more about this?

dnfield · 2020-10-21T16:01:43Z

packages/integration_test/pubspec.yaml

  flutter_test:
    sdk: flutter
  path: ^1.6.4
+  test_core:


@jakemac53 @natebosch - LGTY?

This can be updated now

Suggested change

test_core:

test_core: 0.3.12-nullsafety.6

Unfortunately null safety leaves us in a state where we'll need to pin to a single version since we can't really pin to a feature version currently.

Thanks, I updated it but there's a conflict with flutter_test that is pinned to the previous version. I have opened flutter/flutter#68771 to update that separately.

dnfield

This is pretty exciting - this will fix several classes of problems users can run into currently with the package!

jiahaog · 2020-10-22T12:50:38Z

Still pending flutter/flutter#68771 for CI to pass, but this is ready for review now.

jiahaog · 2020-11-09T07:42:32Z

package:integration_test is being moved to the SDK repo, so this PR is superseded by flutter/flutter#70075

google-cla bot added the cla: yes label Oct 21, 2020

jiahaog commented Oct 21, 2020

View reviewed changes

packages/integration_test/example/pubspec.yaml Outdated Show resolved Hide resolved

packages/integration_test/pubspec.yaml Outdated Show resolved Hide resolved

Remove print

f75bbb4

jiahaog requested a review from dnfield October 21, 2020 13:22

dnfield reviewed Oct 21, 2020

View reviewed changes

packages/integration_test/lib/integration_test.dart Outdated Show resolved Hide resolved

dnfield reviewed Oct 21, 2020

View reviewed changes

packages/integration_test/lib/integration_test.dart Show resolved Hide resolved

dnfield reviewed Oct 21, 2020

View reviewed changes

packages/integration_test/lib/integration_test.dart Outdated Show resolved Hide resolved

dnfield reviewed Oct 21, 2020

View reviewed changes

flutter deleted a comment from natebosch Oct 22, 2020

Update comments, revamp tests

d1891b5

jiahaog marked this pull request as ready for review October 22, 2020 12:49

Revert ios build artifact changes

4dd0d26

jiahaog requested a review from natebosch October 22, 2020 13:43

jiahaog added 2 commits October 27, 2020 06:32

Merge branch 'master' into integration-test-reporter

c5d5492

Bump test dep

2da4ba2

jiahaog mentioned this pull request Nov 9, 2020

[integration_test] Add a run method for proper reporting of test results flutter/flutter#70075

Merged

1 task

jiahaog closed this Nov 9, 2020

jiahaog deleted the integration-test-reporter branch November 9, 2020 07:42

		import 'package:test_core/src/runner/engine.dart';
		import 'package:test_core/src/runner/reporter.dart';

[integration_test] Add a run method for proper reporting of test results #3179

[integration_test] Add a run method for proper reporting of test results #3179

Uh oh!

Conversation

jiahaog commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Example usage

Related Issues

Breaking Change

Uh oh!

jiahaog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiahaog Oct 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnfield left a comment

Choose a reason for hiding this comment

Uh oh!

jiahaog commented Oct 22, 2020

Uh oh!

jiahaog commented Nov 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[integration_test] Add a `run` method for proper reporting of test results #3179

[integration_test] Add a `run` method for proper reporting of test results #3179

jiahaog commented Oct 21, 2020 •

edited

Loading

jiahaog Oct 22, 2020 •

edited

Loading

jiahaog commented Nov 9, 2020 •

edited

Loading