Reporting script. by octalmage · Pull Request #14 · WordPress/phpunit-test-runner

octalmage · 2017-06-28T16:32:49Z

This PR includes the reporting script. It downloads the junit.xml results using rsync, gets the SVN revision, then processes and uploads the junit.xml as json.

Things that still need to be done:

Handling errors like we do failures and adding them to the JSON.
Authentication and attaching the results to a host. One idea: If we use application passwords, the username/password would be tied to the host which we could then attach to the results on the API side.
Capturing anything else we want to be submitted.

The json only includes the failures and it looks something like this:

{
	"tests": "96",
	"failures": "5",
	"errors": "0",
	"testsuites": {
		"Tests_Meta": {
			"name": "Tests_Meta",
			"tests": "18",
			"failures": "1",
			"errors": "0",
			"testcases": {
				"test_user_metadata_not_exists": {
					"name": "test_user_metadata_not_exists",
					"failure": "Tests_Meta::test_user_metadata_not_existsnFailed asserting that 8 matches expected 2.nn/home/public/test/tests/phpunit/tests/meta.php:149n"
				}
			}
		},
		"Tests_Meta_DeleteMetadata": {
			"name": "Tests_Meta_DeleteMetadata",
			"tests": "8",
			"failures": "4",
			"errors": "0",
			"testcases": {
				"test_with_falsey_meta_value_should_not_delete_all_meta": {
					"name": "test_with_falsey_meta_value_should_not_delete_all_meta",
					"failure": "Tests_Meta_DeleteMetadata::test_with_falsey_meta_value_should_not_delete_all_metanFailed asserting that two arrays are equal.n--- Expectedn+++ Actualn@@ @@n Array (n 0 =&gt; '0'n- 1 =&gt; '1'n- 2 =&gt; '2'n+ 1 =&gt; '0'n+ 2 =&gt; '1'n+ 3 =&gt; '2'n+ 4 =&gt; '2'n )nn/home/public/test/tests/phpunit/includes/testcase.php:514n/home/public/test/tests/phpunit/tests/meta/deleteMetadata.php:43n"
				},
				"test_meta_value_should_be_ignored_when_empty_string": {
					"name": "test_meta_value_should_be_ignored_when_empty_string",
					"failure": "Tests_Meta_DeleteMetadata::test_meta_value_should_be_ignored_when_empty_stringnFailed asserting that two arrays are equal.n--- Expectedn+++ Actualn@@ @@n Array (n 0 =&gt; ''n 1 =&gt; '0'n- 2 =&gt; '1'n- 3 =&gt; '2'n+ 2 =&gt; '0'n+ 3 =&gt; '0'n+ 4 =&gt; '1'n+ 5 =&gt; '1'n+ 6 =&gt; '2'n+ 7 =&gt; '2'n+ 8 =&gt; '2'n )nn/home/public/test/tests/phpunit/includes/testcase.php:514n/home/public/test/tests/phpunit/tests/meta/deleteMetadata.php:62n"
				},
				"test_meta_value_should_be_ignored_when_null": {
					"name": "test_meta_value_should_be_ignored_when_null",
					"failure": "Tests_Meta_DeleteMetadata::test_meta_value_should_be_ignored_when_nullnFailed asserting that two arrays are equal.n--- Expectedn+++ Actualn@@ @@n Array (n 0 =&gt; ''n- 1 =&gt; '0'n- 2 =&gt; '1'n- 3 =&gt; '2'n+ 1 =&gt; ''n+ 2 =&gt; '0'n+ 3 =&gt; '0'n+ 4 =&gt; '0'n+ 5 =&gt; '0'n+ 6 =&gt; '1'n+ 7 =&gt; '1'n+ 8 =&gt; '1'n+ 9 =&gt; '2'n+ 10 =&gt; '2'n+ 11 =&gt; '2'n+ 12 =&gt; '2'n )nn/home/public/test/tests/phpunit/includes/testcase.php:514n/home/public/test/tests/phpunit/tests/meta/deleteMetadata.php:78n"
				},
				"test_meta_value_should_be_ignored_when_false": {
					"name": "test_meta_value_should_be_ignored_when_false",
					"failure": "Tests_Meta_DeleteMetadata::test_meta_value_should_be_ignored_when_falsenFailed asserting that two arrays are equal.n--- Expectedn+++ Actualn@@ @@n Array (n 0 =&gt; ''n- 1 =&gt; '0'n- 2 =&gt; '1'n- 3 =&gt; '2'n+ 1 =&gt; ''n+ 2 =&gt; ''n+ 3 =&gt; '0'n+ 4 =&gt; '0'n+ 5 =&gt; '0'n+ 6 =&gt; '0'n+ 7 =&gt; '0'n+ 8 =&gt; '1'n+ 9 =&gt; '1'n+ 10 =&gt; '1'n+ 11 =&gt; '1'n+ 12 =&gt; '2'n+ 13 =&gt; '2'n+ 14 =&gt; '2'n+ 15 =&gt; '2'n+ 16 =&gt; '2'n )nn/home/public/test/tests/phpunit/includes/testcase.php:514n/home/public/test/tests/phpunit/tests/meta/deleteMetadata.php:94n"
				}
			}
		},
		"Tests_Meta_Query": {
			"name": "Tests_Meta_Query",
			"tests": "36",
			"failures": "0",
			"errors": "0",
			"testcases": []
		},
		"Tests_Meta_Register_Meta": {
			"name": "Tests_Meta_Register_Meta",
			"tests": "25",
			"failures": "0",
			"errors": "0",
			"testcases": []
		},
		"Tests_Meta_Slashes": {
			"name": "Tests_Meta_Slashes",
			"tests": "7",
			"failures": "0",
			"errors": "0",
			"testcases": []
		},
		"Tests_Meta_UpdateMetadata": {
			"name": "Tests_Meta_UpdateMetadata",
			"tests": "2",
			"failures": "0",
			"errors": "0",
			"testcases": []
		}
	}
}

danielbachhuber · 2017-06-28T17:06:08Z

report.php

+	log_message( 'Error uploading results' );
+}
+
+function process( $path )


Can we put this in functions.php ?

Sure! Wasn't sure if we should put single use functions there, but I'll move it.

👍 Sounds good. It probably could use a slightly more verbose name too.

danielbachhuber

Few things I noticed

danielbachhuber · 2017-07-19T16:50:31Z

functions.php

+ * @return string
+ */
+function get_wordpress_version( $dir ) {
+	$wpVersion = file_get_contents( trailingslashit( $dir ) . 'src/wp-includes/version.php' );


Don't we want the actual changeset reported with the WordPress version? While this WordPress version includes a changeset, it doesn't get incremented very often so it's likely to be incorrect.

That's a good point, this won't get updated on commit. We get the SVN revision, so we should be able to get the changeset from that. Not sure if this will be actually useful.

danielbachhuber · 2017-07-19T16:52:22Z

report.php

+	$junit_location = '-e "ssh ' . $WPT_SSH_OPTIONS . '" ' . escapeshellarg( $WPT_SSH_CONNECT . ':' . $junit_location );
+}
+
+$junit_exec = 'rsync -rv ' . $junit_location . ' ./';


Can we copy this to $WPT_PREPARE_DIR for easy consistency?

danielbachhuber · 2017-07-19T16:52:52Z

report.php

+$results = process_junit_xml( $xml );
+
+$meta = array(
+	'php_version' => phpversion(),


Doesn't this need to be the phpversion() on the remote filesystem?

Yes it does, we'll need to find a way to get that, probably during the testing process.

Added in #26 (comment)

The env.json file ends up in the same directory as junit.xml to make it easy to sync over.

danielbachhuber · 2017-07-19T16:53:31Z

report.php

+if ( $success ) {
+	log_message( 'Results successfully uploaded' );
+} else {
+	log_message( 'Error uploading results' );


Can we use the error_message() helper here?

danielbachhuber · 2017-07-19T16:54:42Z

report.php

+$WPT_REPORT_API_KEY = getenv( 'WPT_REPORT_API_KEY' );
+
+log_message('Getting SVN Revision');
+$rev = exec('git -C ' . escapeshellarg( $WPT_PREPARE_DIR ) . ' log -1 --pretty=%B | grep "git-svn-id:" | cut -d " " -f 2 | cut -d "@" -f 2');


Can we use the perform_operations() helper?

perform_operations doesn't appear to return the output, so we can't get the revision from git.

danielbachhuber · 2017-07-19T16:55:46Z

cleanup.php

 perform_operations( array(
 	'rm -rf ' . escapeshellarg( $WPT_PREPARE_DIR . '/.git' ),
 	'rm -r ' . escapeshellarg( $WPT_PREPARE_DIR ),
+	'rm  ./junit.xml',


Ideally, we'd put this in the $WPT_PREPARE_DIR to reduce complexity.

danielbachhuber · 2017-07-19T17:13:02Z

composer.json

+  "repositories": [
+    {
+      "type": "git",
+      "url": "https://github.com/octalmage/wp-unit-test-api-client-php.git"


Does this really need to be a separate library, or can we consolidate into this repo?

For simplicity's sake, it'd be better to keep everything together if we can.

octalmage · 2017-07-26T15:52:05Z

@danielbachhuber made the requested changes, and removed the external dependency.

Let me know what you think!

danielbachhuber · 2017-07-26T17:09:26Z

report.php

+	log_message( 'Results successfully uploaded' );
+} else {
+	error_message( 'Error uploading results' );
+}


It'd be good to enrich this reporting, but it doesn't need to block merge #27

octalmage added 4 commits June 28, 2017 11:10

Add vendor to .gitignore.

8037549

Add composer with wp-unit-test-api-client-php.

43fb90e

Add reporting script.

15943be

Add cleanup.

303999f

danielbachhuber reviewed Jun 28, 2017

View reviewed changes

octalmage added 3 commits June 29, 2017 16:56

Use API key to submit results.

9f64b92

Update API client.

67f6c0f

Remove extra line.

060d10e

octalmage force-pushed the report branch from b1944b8 to b7e8607 Compare June 29, 2017 22:44

Pass meta data.

416a172

octalmage force-pushed the report branch from b7e8607 to 416a172 Compare June 29, 2017 22:45

Update env.default with API key format.

6492a6c

danielbachhuber requested changes Jul 19, 2017

View reviewed changes

octalmage added 2 commits July 26, 2017 10:12

Copy files down to WPT_PREPARE_DIR and load env.json as meta.

f528a45

Use local upload function.

0388b24

danielbachhuber mentioned this pull request Jul 26, 2017

Log results of reporting API interactions #27

Closed

danielbachhuber reviewed Jul 26, 2017

View reviewed changes

danielbachhuber approved these changes Jul 26, 2017

View reviewed changes

danielbachhuber merged commit 41d16fa into master Jul 26, 2017

danielbachhuber deleted the report branch July 26, 2017 17:10

Conversation

octalmage commented Jun 28, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielbachhuber left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

octalmage commented Jul 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants