Skip to content

test: auto-throttle admin requests #4178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
lesv merged 1 commit into from
Nov 6, 2020
Merged

test: auto-throttle admin requests #4178

lesv merged 1 commit into from
Nov 6, 2020

Conversation

@olavloite
Copy link
Contributor

@olavloite olavloite commented Nov 5, 2020

The Spanner client has the option to automatically throttle the number of administrative requests that are generated in order to avoid hitting the hard limit that Spanner has of max 5 admin requests per second.

Updates #4144

@olavloite olavloite requested a review from a team November 5, 2020 10:53
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 5, 2020
@snippet-bot
Copy link

snippet-bot bot commented Nov 5, 2020

Here is the summary of changes.

You added 1 region tag.

@olavloite olavloite mentioned this pull request Nov 5, 2020
Merged
@lesv
Copy link
Contributor

lesv commented Nov 5, 2020

Java 11 failed. I'm a bit surprised you left setAutoThrottleAdministrativeRequests() out of what's shown users. It's probably not a bad idea to tell users about it.

@olavloite
Copy link
Contributor Author

olavloite commented Nov 5, 2020

Java 11 failed. I'm a bit surprised you left setAutoThrottleAdministrativeRequests() out of what's shown users. It's probably not a bad idea to tell users about it.

I agree, but I think that should be in a separate sample. This specific sample is for creating a basic Spanner client, and this option should really not be necessary for most users.

@lesv
Copy link
Contributor

lesv commented Nov 5, 2020

Java 11

Well, nothing like being bitten by the very thing you are trying to fix.
I can force this to merge if it's ok w/ you.

------------------------------------------------------------
- testing spanner/cloud-client
------------------------------------------------------------
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 550.532 s <<< FAILURE! - in com.example.spanner.SpannerSampleIT
[ERROR] com.example.spanner.SpannerSampleIT.testSample  Time elapsed: 532.472 s  <<< ERROR!
com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Operation with name "projects/java-docs-samples-testing/instances/default-instance/databases/restored-d53e1df9-96e2-448c-b/operations/_auto_op_bc27f7f77a9eef1c" failed with status = GrpcStatusCode{transportCode=RESOURCE_EXHAUSTED} and message = The instance projects/java-docs-samples-testing/instances/default-instance has too many database splits to complete the operation.  Please add more nodes and try again.
	at com.example.spanner.SpannerSampleIT.runSample(SpannerSampleIT.java:61)
	at com.example.spanner.SpannerSampleIT.testSample(SpannerSampleIT.java:315)
Caused by: com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Operation with name "projects/java-docs-samples-testing/instances/default-instance/databases/restored-d53e1df9-96e2-448c-b/operations/_auto_op_bc27f7f77a9eef1c" failed with status = GrpcStatusCode{transportCode=RESOURCE_EXHAUSTED} and message = The instance projects/java-docs-samples-testing/instances/default-instance has too many database splits to complete the operation.  Please add more nodes and try again.

[ERROR] Errors:
[ERROR]   SpannerSampleIT.testSample:315->runSample:61 » Spanner RESOURCE_EXHAUSTED: Ope...
[ERROR] Tests run: 16, Failures: 0, Errors: 1, Skipped: 0

@lesv
Copy link
Contributor

lesv commented Nov 5, 2020

@averikitsch @stephaniewang526 Care to give feedback on a force merge of this fix? (ie. if you disagree, I won't do it)

@averikitsch
Copy link
Contributor

averikitsch commented Nov 5, 2020

Do we exhaust resources regularly?

@lesv
Copy link
Contributor

lesv commented Nov 5, 2020

Recently, we have started. It's affecting 4 PR's ATM.

@averikitsch
Copy link
Contributor

averikitsch commented Nov 5, 2020

If we don't mind accepting this error as a passing test we should add it to the test code

@lesv
Copy link
Contributor

lesv commented Nov 5, 2020

The problem is that we have other tests that are running that are wiping out the quota - by accepting this, it will start to clear.

@averikitsch
Copy link
Contributor

averikitsch commented Nov 5, 2020

I'm fine force merging

@lesv lesv merged commit 91b92f7 into GoogleCloudPlatform:master Nov 6, 2020
@olavloite olavloite deleted the spanner-limit-administrative-requests branch November 6, 2020 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@lesv lesv lesv approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@stephaniewang526 stephaniewang526

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@olavloite @lesv @averikitsch @stephaniewang526