Use fixture to test the repository-gcs plugin by tlrx · Pull Request #28788 · elastic/elasticsearch

tlrx · 2018-02-22T16:24:38Z

This commit adds a GoogleCloudStorageFixture that uses the logic of a GoogleCloudStorageTestServer (added in #28576) to emulate a remote Google Cloud Storage service.

By adding this fixture and a new integration test, we should be able to catch more bugs when upgrading the client library. In consequence the old REST test RepositoryGcsClientYamlTestSuiteIT has been removed. Sadly the GoogleCloudStorageTestServer has to be adapted for a better handling of batch requests. The fixture is started by the googleCloudStorageFixture task and a custom Service Account file is created and added to the Elasticsearch keystore before the integration test.

The fixture resides in the repository-gcs plugin project.

This commit adds a GoogleCloudStorageFixture that uses the logic of a GoogleCloudStorageTestServer (added in elastic#28576) to emulate a remote Google Cloud Storage service. By adding this fixture and a new integration test, we should be able to catch more bugs when upgrading the client library. In consequence the old REST test RepositoryGcsClientYamlTestSuiteIT has been removed. The GoogleCloudStorageTestServer has to be adapted a bit for batch requests. The fixture is started by the googleCloudStorageFixture task and a custom Service Account file is created and added to the Elasticsearch keystore for each test.

rjernst

This is great @tlrx! Just a few suggestions.

rjernst · 2018-02-25T22:03:44Z

plugins/repository-gcs/build.gradle

+}
+
+/** A service account file that points to the Google Cloud Storage service emulated by the fixture **/
+File serviceAccountFile = new File(project.buildDir, "/generated-resources/service_account_test.json")


The first / is not necessary.

rjernst · 2018-02-26T00:16:21Z

plugins/repository-gcs/build.gradle

+            '  "private_key": "-----BEGIN PRIVATE KEY-----\\n' + encodedKey + '\\n-----END PRIVATE KEY-----\\n",\n' +
+            '  "client_email": "integration_test@appspot.gserviceaccount.com",\n' +
+            '  "client_id": "123456789101112130594",\n' +
+            "  \"auth_uri\": \"http://${ -> googleCloudStorageFixture.addressAndPort }/o/oauth2/auth\",\n" +


I don't think the closures are necessary here, since setText takes String and thus these will immediately be resolved.

rjernst · 2018-02-26T00:27:16Z

...ns/repository-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageIT.java

+import static org.apache.http.entity.ContentType.APPLICATION_JSON;
+import static org.hamcrest.Matchers.containsString;
+
+public class GoogleCloudStorageIT extends ESRestTestCase {


Is there anything that prevents this from still being in a yaml test?

Not really. I thought we reserved YAML tests to test the REST API.

I reverted this change and added back the yaml test in a more complete fashion.

rjernst · 2018-02-26T00:28:33Z

...itory-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageTestServer.java

     */
-    GoogleCloudStorageTestServer(final String endpoint, final boolean prefixWithEndpoint) {
-        this.handlers = defaultHandlers(endpoint, prefixWithEndpoint, buckets);
+    GoogleCloudStorageTestServer(final String endpoint) {


Long term (not in this PR) I think it would be good to move this into the fixture, and use a more pure unit testing mock for the service calls inside the unit tests.

+1

I created the issue #28960 to track this.

tlrx · 2018-03-09T12:58:57Z

Thanks @rjernst

This commit adds a GoogleCloudStorageFixture that uses the logic of a GoogleCloudStorageTestServer (added in #28576) to emulate a remote Google Cloud Storage service. By adding this fixture and a more complete integration test, we should be able to catch more bugs when upgrading the client library. The fixture is started by the googleCloudStorageFixture task and a custom Service Account file is created and added to the Elasticsearch keystore for each test.

tlrx · 2018-03-09T17:03:34Z

This change has been backported to 6.x in be8ae87

This commit adds a GoogleCloudStorageFixture that uses the logic of a GoogleCloudStorageTestServer (added in elastic#28576) to emulate a remote Google Cloud Storage service. By adding this fixture and a more complete integration test, we should be able to catch more bugs when upgrading the client library. The fixture is started by the googleCloudStorageFixture task and a custom Service Account file is created and added to the Elasticsearch keystore for each test.

* master: (28 commits) Maybe die before failing engine (elastic#28973) Remove special handling for _all in nodes info Remove Booleans use from XContent and ToXContent (elastic#28768) Update Gradle Testing Docs (elastic#28970) Make primary-replica resync failures less lenient (elastic#28534) Remove temporary file 10_basic.yml~ Use different pipeline id in test. (pipelines do not get removed between tests extending from ESIntegTestCase) Use fixture to test the repository-gcs plugin (elastic#28788) Use String.join() to describe a list of tasks (elastic#28941) Fixed incorrect test try-catch statement Plugins: Consolidate plugin and module loading code (elastic#28815) percolator: Take `matchAllDocs` and `verified` of the sub result into account when analyzing a function_score query. Build: Remove rest tests on archive distribution projects (elastic#28952) Remove FastStringReader in favor of vanilla StringReader (elastic#28944) Remove FastCharArrayReader and FastCharArrayWriter (elastic#28951) Continue registering pipelines after one pipeline parse failure. (elastic#28752) Build: Fix ability to ignore when no tests are run (elastic#28930) [rest-api-spec] update doc link for /_rank_eval Switch XContentBuilder from BytesStreamOutput to ByteArrayOutputStream (elastic#28945) Factor UnknownNamedObjectException into its own class (elastic#28931) ...

This commit adds a new fixture that emulates a S3 service in order to improve the existing integration tests. This is very similar to what has been made for Google Cloud Storage in elastic#28788, and such tests would have helped a lot to catch bugs like elastic#22534. The AmazonS3Fixture is brittle and only implements the very necessary stuff for the S3 repository to work, but at least it works and can be adapted for specific tests needs.

This commit adds a new fixture that emulates a S3 service in order to improve the existing integration tests. This is very similar to what has been made for Google Cloud Storage in #28788, and such tests would have helped a lot to catch bugs like #22534. The AmazonS3Fixture is brittle and only implements the very necessary stuff for the S3 repository to work, but at least it works and can be adapted for specific tests needs.

This commit adds a new fixture that emulates an Azure Storage service in order to improve the existing integration tests. This is very similar to what has been made for Google Cloud Storage in elastic#28788 and for Amazon S3 in elastic#29296, and it would have helped a lot to catch bugs like elastic#22534.

This commit adds a new fixture that emulates an Azure Storage service in order to improve the existing integration tests. This is very similar to what has been made for Google Cloud Storage in #28788 and for Amazon S3 in #29296, and it would have helped a lot to catch bugs like #22534.

Similarly to what has been done in for the repository-s3 plugin, this commit moves the fixture test into a dedicated repository-gcs/qa/google-cloud-storage project. It also exposes some environment variables which allows to execute the integration tests against the real Google Cloud Storage service. When the environment variables are not defined, the integration tests are executed using the fixture added in #28788. Related to #29349.

tlrx added >test Issues or PRs that are addressing/adding tests review :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v7.0.0 labels Feb 22, 2018

tlrx requested review from rjernst and ywelsch February 22, 2018 16:24

rjernst approved these changes Feb 26, 2018

View reviewed changes

Apply feedback

ac30d03

tlrx mentioned this pull request Mar 9, 2018

Use unit testing mock for the service calls in repository-gcs #28960

Closed

tlrx merged commit 4756790 into elastic:master Mar 9, 2018

tlrx deleted the use-fixture-for-repository-gcs branch March 9, 2018 12:58

tlrx added the v6.3.0 label Mar 9, 2018

tlrx mentioned this pull request Mar 23, 2018

Create S3 repository tests that test against a real S3 bucket #20862

Closed

tlrx mentioned this pull request Mar 29, 2018

Use fixture to test repository-s3 plugin #29296

Merged

tlrx mentioned this pull request Apr 3, 2018

Use fixture to test repository-azure plugin #29347

Merged

tlrx mentioned this pull request Apr 3, 2018

Test cloud based repository plugins using real services backends #29349

Closed

3 tasks

tlrx mentioned this pull request Apr 27, 2018

Move repository-gcs fixture test to QA project and allow testing against external service #30208

Merged

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use fixture to test the repository-gcs plugin#28788

Use fixture to test the repository-gcs plugin#28788
tlrx merged 2 commits intoelastic:masterfrom
tlrx:use-fixture-for-repository-gcs

tlrx commented Feb 22, 2018 •

edited

Loading

Uh oh!

rjernst left a comment

Uh oh!

rjernst Feb 25, 2018

Uh oh!

rjernst Feb 26, 2018

Uh oh!

rjernst Feb 26, 2018

Uh oh!

tlrx Mar 9, 2018

Uh oh!

rjernst Feb 26, 2018

Uh oh!

tlrx Mar 9, 2018

Uh oh!

tlrx commented Mar 9, 2018

Uh oh!

tlrx commented Mar 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tlrx commented Feb 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

rjernst Feb 25, 2018

Choose a reason for hiding this comment

Uh oh!

rjernst Feb 26, 2018

Choose a reason for hiding this comment

Uh oh!

rjernst Feb 26, 2018

Choose a reason for hiding this comment

Uh oh!

tlrx Mar 9, 2018

Choose a reason for hiding this comment

Uh oh!

rjernst Feb 26, 2018

Choose a reason for hiding this comment

Uh oh!

tlrx Mar 9, 2018

Choose a reason for hiding this comment

Uh oh!

tlrx commented Mar 9, 2018

Uh oh!

tlrx commented Mar 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tlrx commented Feb 22, 2018 •

edited

Loading