test: delete stale test databases #4133

olavloite · 2020-11-03T16:49:29Z

Delete stale test databases that have the same name as those being created by this test.

spanner/cloud-client/src/test/java/com/example/spanner/SpannerSampleIT.java

kurtisvg · 2020-11-04T17:46:09Z

spanner/cloud-client/src/test/java/com/example/spanner/SpannerSampleIT.java

+    for (Database db : dbClient.listDatabases(instanceId).iterateAll()) {
+      if (db.getCreateTime().getSeconds() > 0 && TimeUnit.HOURS
+          .convert(now.getSeconds() - db.getCreateTime().getSeconds(), TimeUnit.SECONDS) > 24) {
+        if (pattern.matcher(toComparableId(baseDbId, db.getId().getDatabase())).matches()) {


Would it be less fragile to just use:

if db.getId.getDatabase().startsWith(baseDbID)?

Otherwise any adjustments to the formatForTest UUID will break this without any immediate feedback.

Well, yes, that would be possible, but I do worry about doing that: The method will delete all databases on the instance that meet the criteria, so if someone were to execute these tests on a random instance and the spanner.sample.database happens to be set to a fairly generic value (something like 'test' or 'sample'), it could potentially delete more databases than it should.

This is also only intended to be a fallback in case of unexpected test failures. The databases that are created by the tests are automatically deleted at the end of the test run, unless the test run is killed.

@kurtisvg Could you supply me with a list of the IDs of the databases that are currently present on the test instance? The build on Java 11 claims to be successful. I would have expected that to have deleted all stale test databases on the instance, but we are currently seeing a lot of build failures that seem to be caused by too many databases still being present on that instance. See for example #4175.

Project: Java-docs-samples-testing

Region: us-central1

ID: default-instance w/ 91 databases

bjfjf-e6ce3e71-2221-4000-a | 0.12% | 92.96 KB | | df-spanner-read-it-292649216 | 0.21% | 0 B | | jfodx-f432c184-5c6b-4d14-b | 0.16% | 80.46 KB | | mysample-09631a0d-0107-4fc5-a | 0.09% | 1.81 KB | | mysample-0b340a88-d096-4261-9 | 0.16% | 1.81 KB | | mysample-13337208-538b-4392-b | 0.12% | 1.81 KB | | mysample-14472e40-9349-4b8f-9 | 0.10% | 1.81 KB | | mysample-17263438-c62c-4ef0-9 | 0.21% | 1.81 KB | | mysample-17caeba3-ca10-4808-8 | 0.12% | 1.81 KB | | mysample-1a13079a-c6d6-4694-a | 0.21% | 212 B | | mysample-1d4c2207-f1aa-4556-a | 0.17% | 1.08 KB | | mysample-2878ac26-95f0-4348-b | 0.19% | 1.1 KB | | mysample-2b863578-1bab-410c-b | 0.10% | 1.81 KB | | mysample-32dc29c8-f074-4bdb-9 | 0.15% | 1.81 KB | | mysample-3d64a361-75da-4ea7-9 | 0.12% | 1.81 KB | | mysample-439adf53-4419-46ae-9 | 0.13% | 1.81 KB | | mysample-476db395-aa6e-4624-9 | 0.16% | 1.81 KB | | mysample-486d8bac-4077-4c11-8 | 0.09% | 1.81 KB | | mysample-4b940c0b-fffc-4caa-a | 0.17% | 1.81 KB | | mysample-4ca24702-bd50-41a4-a | 0.13% | 474 B | | mysample-4ceafb69-cd43-492a-b | 0.11% | 1.81 KB | | mysample-50918327-f321-4516-8 | 0.17% | 762 B | | mysample-6635ddaf-2bda-43ed-b | 0.11% | 1.81 KB | | mysample-6f48aa6b-159a-4a7a-8 | 0.23% | 1.81 KB | | mysample-70ea21c8-4a55-40d0-a | 0.10% | 732 B | | mysample-73af71b0-b212-4794-b | 0.10% | 603 B | | mysample-7819d309-ef65-4eab-a | 0.11% | 1.81 KB | | mysample-7cbe0849-f091-437e-9 | 0.14% | 1.81 KB | | mysample-7ead3ac1-f28a-43b5-8 | 0.18% | 729 B | | mysample-807e0e6a-9afd-46da-b | 0.08% | 1.81 KB | | mysample-82dc45c0-952d-467b-a | 0.10% | 1.81 KB | | mysample-84e5b267-4ff7-409e-8 | 0.08% | 1.81 KB | | mysample-860840f6-e13c-499d-a | 0.16% | 1.81 KB | | mysample-8aabfc92-4838-479b-b | 0.18% | 0 B | | mysample-90037d1b-cc70-4f75-b | 0.19% | 1.81 KB | | mysample-9dbe20a4-d4a0-49a7-a | 0.14% | 1.43 KB | | mysample-ad1162e9-205b-4274-8 | 0.13% | 550 B | | mysample-af602369-6897-445e-9 | 0.12% | 651 B | | mysample-b22cc310-fd3c-4db0-b | 0.12% | 1.4 KB | | mysample-bb4fb4cd-7317-4b31-9 | 0.43% | 708 B | | mysample-c22adce8-0466-4781-9 | 0.13% | 1.81 KB | | mysample-c62bcee2-17b3-4423-9 | 0.08% | 1.81 KB | | mysample-c8fdfb9b-021f-4ab1-8 | 0.11% | 1.81 KB | | mysample-d744d126-1405-4a66-a | 0.12% | 1.81 KB | | mysample-daf852ef-98f9-4d54-a | 0.19% | 815 B | | mysample-def739ce-d1ce-4e5e-a | 0.10% | 1.81 KB | | mysample-e185f237-e813-4cbf-b | 0.17% | 1.81 KB | | mysample-ebd1c05e-8ddf-420b-b | 0.22% | 354 B | | mysample-fce9edf0-0bb2-4d5c-b | 0.12% | 1.81 KB | | occvu-0a80960f-697d-4609-b | 0.14% | 97.46 KB | | quickstart-db | 0.14% | 89 B | | restored-004abec7-52a9-4918-a | 0.12% | 1.86 KB | | restored-0b340a88-d096-4261-9 | 0.12% | 1.81 KB | | restored-12bd398b-0972-4ab4-b | 0.12% | 1.86 KB | | restored-1d4c2207-f1aa-4556-a | 0.11% | 1.86 KB | | restored-213afcab-89df-4219-a | 0.24% | 1.86 KB | | restored-2878ac26-95f0-4348-b | 0.15% | 1.86 KB | | restored-2aafc9b2-72ab-4aec-b | 0.19% | 1.86 KB | | restored-2b863578-1bab-410c-b | 0.11% | 1.81 KB | | restored-2f9a1547-fe1e-48f3-9 | 0.11% | 1.86 KB | | restored-3060273a-5185-470a-8 | 0.08% | 1.86 KB | | restored-3bb24fe6-7fb7-44af-b | 0.15% | 1.86 KB | | restored-4b05449a-4c8f-4769-b | 0.16% | 1.86 KB | | restored-4b2de289-6000-463d-8 | 0.07% | 1.86 KB | | restored-50918327-f321-4516-8 | 0.24% | 1.86 KB | | restored-55e86d1a-b7cd-4af2-b | 0.11% | 1.86 KB | | restored-61a6eb6b-b9a6-4b65-a | 0.12% | 1.86 KB | | restored-77c6e8b1-f5b3-4ca8-a | 0.12% | 1.86 KB | | restored-790f8c0f-3106-4d53-8 | 0.13% | 1.86 KB | | restored-79a3039b-9774-4ad3-a | 0.16% | 1.86 KB | | restored-7cbe0849-f091-437e-9 | 0.09% | 1.81 KB | | restored-7e1e2fdd-4e75-4c2c-8 | 0.12% | 1.86 KB | | restored-7ead3ac1-f28a-43b5-8 | 0.32% | 1.86 KB | | restored-8055ce7a-b316-4414-9 | 0.10% | 1.86 KB | | restored-82dc45c0-952d-467b-a | 0.10% | 1.81 KB | | restored-9dbe20a4-d4a0-49a7-a | 0.25% | 1.86 KB | | restored-a90ec3d8-9d3e-4f89-b | 0.12% | 1.86 KB | | restored-af602369-6897-445e-9 | 0.17% | 1.86 KB | | restored-afcf21cf-e927-4c6b-9 | 0.15% | 1.86 KB | | restored-bb4fb4cd-7317-4b31-9 | 0.09% | 1.86 KB | | restored-c22adce8-0466-4781-9 | 0.14% | 1.81 KB | | restored-c62bcee2-17b3-4423-9 | 0.11% | 1.81 KB | | restored-c8fdfb9b-021f-4ab1-8 | 0.13% | 1.81 KB | | restored-d744d126-1405-4a66-a | 0.11% | 1.81 KB | | restored-daf852ef-98f9-4d54-a | 0.12% | 1.86 KB | | restored-e185f237-e813-4cbf-b | 0.12% | 1.81 KB | | restored-mysample-16ae8df9-7ec | 0.10% | 1.81 KB | | sejzn-3d2384a5-1344-4bff-8 | 0.18% | 18.46 KB | | vzdqp-da80840c-38c5-430b-a | 0.16% | 65.96 KB | | wfwyd-f251339f-d247-4031-b | 0.14% | 24.46 KB | | zfidi-cecbc595-7e56-4ee9-9 | 0.07% | 109.46 KB |

There is also samples-926dcd35-e183-4b8c-a in asia w/ 0 databases.

Maybe it would be better to remove this logic from the test all together, and instead just put it in a Cloud Function that runs periodically to clean up stale resources in the project.

@lesv could you delete the existing databases? We are seeing errors when running the samples test in our other repository as well (we are in the process of migrating the samples from this repository to the other one - https://github.com/googleapis/java-spanner/tree/master/samples)

There is also samples-926dcd35-e183-4b8c-a in asia w/ 0 databases.

As far as I'm concerned, that one may be deleted. I can't see that it is used by anything in the Java samples.

Thanks for the list of the existing databases. That helped a lot.

@kurtisvg PTAL

lesv · 2020-11-04T22:35:20Z

Java 8 is failing

olavloite · 2020-11-05T10:16:28Z

Java 8 is failing

It fails with the following error:

com.google.cloud.spanner.SpannerException:
RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute'

skuruppu · 2020-11-05T10:32:48Z

Java 8 is failing

It fails with the following error:

com.google.cloud.spanner.SpannerException:
RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute'

For those, we were asked by the Spanner team to retry the tests. In Node.js, we retry the admin requests with some exponential backoff to get around this problem.

olavloite · 2020-11-05T10:54:53Z

Java 8 is failing

It fails with the following error:
com.google.cloud.spanner.SpannerException:
RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' 
For those, we were asked by the Spanner team to retry the tests. In Node.js, we retry the admin requests with some exponential backoff to get around this problem.

The Java client has a built-in option to throttle the number of admin requests. See #4178

thiagotnunes

We will need to do this fix in https://github.com/googleapis/java-spanner/tree/master/samples, since we are in the progress of moving samples to that repository.

test: delete stale test databases

e851b3d

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 3, 2020

olavloite added 3 commits November 4, 2020 08:19

test: actually delete databases

c4cffc8

fix: fix NPE

db8c168

fix: only remove stale databases once

0872b64

olavloite marked this pull request as ready for review November 4, 2020 10:51

olavloite requested review from a team, lesv and skuruppu November 4, 2020 10:51

olavloite changed the title ~~[WIP] test: delete stale test databases~~ test: delete stale test databases Nov 4, 2020

blunderbuss-gcf bot assigned kurtisvg Nov 4, 2020

kurtisvg suggested changes Nov 4, 2020

View reviewed changes

fix: reuse baseDbId

0816514

olavloite added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 4, 2020

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 4, 2020

skuruppu requested a review from thiagotnunes November 5, 2020 10:31

lesv approved these changes Nov 6, 2020

View reviewed changes

thiagotnunes approved these changes Nov 6, 2020

View reviewed changes

olavloite added 8 commits November 6, 2020 10:08

fix: also delete stale restored databases

8d14233

test: force logging of non-deleted db

34efac7

fix: log non-deleted databases

008f634

fix: show age in hours

78b593e

fix: drop dbs with length at least 20

49d14ee

fix: drop dbs with length at least 20

dc0ba7a

fix: delete all restored db's that have a long name

619674d

fix: use different patterns for sample and restored

b98b43e

fix: remove deleted check as it is no longer needed

37c8816

olavloite requested a review from kurtisvg November 6, 2020 11:14

kurtisvg approved these changes Nov 6, 2020

View reviewed changes

lesv merged commit f8a99e0 into GoogleCloudPlatform:master Nov 6, 2020

olavloite deleted the spanner-delete-stale-databases branch November 13, 2020 07:20

olavloite mentioned this pull request Nov 13, 2020

fix: delete stale sample databases googleapis/java-spanner#622

Merged

test: delete stale test databases #4133

test: delete stale test databases #4133

Uh oh!

Conversation

olavloite commented Nov 3, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lesv commented Nov 4, 2020

Uh oh!

olavloite commented Nov 5, 2020

Uh oh!

skuruppu commented Nov 5, 2020

Uh oh!

olavloite commented Nov 5, 2020

Uh oh!

thiagotnunes left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants