fix(bigquery): reduce default backoffs#10558
Merged
shollyman merged 3 commits intogoogleapis:mainfrom Jul 17, 2024
Merged
Conversation
This PR reduces the initial backoff used during job polling from 1s to 50ms. In practice most time is spent in the hanging poll on the server side in GetQueryResults, so this reduces opportunities for wasted waiting. Fixes: googleapis#10555
alvarowolfx
reviewed
Jul 17, 2024
| @@ -234,9 +234,9 @@ func runWithRetry(ctx context.Context, call func() error) error { | |||
| func runWithRetryExplicit(ctx context.Context, call func() error, allowedReasons []string) error { | |||
| // These parameters match the suggestions in https://cloud.google.com/bigquery/sla. | |||
Contributor
There was a problem hiding this comment.
Sounds like with those changes, it is not going to match the suggestions found in https://cloud.google.com/bigquery/sla
"Back-off Requirements" means, when an error occurs, the Customer Application is responsible for waiting for a period of time before issuing another request. This means that after the first error, there is a minimum back-off interval of 1 second and for each consecutive error, the back-off interval increases exponentially up to 32 seconds.
...a minimum back-off interval of 1 second...
Contributor
Author
There was a problem hiding this comment.
Ahh yes. I'll back out the retry changes.
alvarowolfx
approved these changes
Jul 17, 2024
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
Jul 22, 2024
🤖 I have created a release *beep* *boop* --- ## [1.62.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.61.0...bigquery/v1.62.0) (2024-07-22) ### Features * **bigquery/analyticshub:** Support Direct Table Access Toggle (Egress GA) ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery/analyticshub:** Support public directory self service for Listings/Exchanges ([#10485](https://togithub.com/googleapis/google-cloud-go/issues/10485)) ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery:** Add rounding mode to FieldSchema ([#10328](https://togithub.com/googleapis/google-cloud-go/issues/10328)) ([1a9e204](https://togithub.com/googleapis/google-cloud-go/commit/1a9e204d7752c5bfe9edfd7bc7ee36c5b1385783)) * **bigquery:** Json support on managedwriter/adapt pkg ([#10542](https://togithub.com/googleapis/google-cloud-go/issues/10542)) ([978d4a1](https://togithub.com/googleapis/google-cloud-go/commit/978d4a1e47cbd8d4bf567b616381a2f12fac4cab)) * **bigquery:** Support column name character map in load jobs ([#10425](https://togithub.com/googleapis/google-cloud-go/issues/10425)) ([b829327](https://togithub.com/googleapis/google-cloud-go/commit/b82932789af82b5e0799c20a096aab98132b5eb1)) ### Bug Fixes * **bigquery/storage/managedwriter:** Faster context failure on send ([#10169](https://togithub.com/googleapis/google-cloud-go/issues/10169)) ([1fb0e64](https://togithub.com/googleapis/google-cloud-go/commit/1fb0e6401d584bf8ede60a170b4d82dc211010b8)) * **bigquery:** Bump google.golang.org/api@v0.187.0 ([8fa9e39](https://togithub.com/googleapis/google-cloud-go/commit/8fa9e398e512fd8533fd49060371e61b5725a85b)) * **bigquery:** Bump google.golang.org/grpc@v1.64.1 ([8ecc4e9](https://togithub.com/googleapis/google-cloud-go/commit/8ecc4e9622e5bbe9b90384d5848ab816027226c5)) * **bigquery:** Empty slice instead of nil slice for primitive repeated fields ([#7315](https://togithub.com/googleapis/google-cloud-go/issues/7315)) ([b371210](https://togithub.com/googleapis/google-cloud-go/commit/b3712100831061fea8605e574d482d7f768ecf14)) * **bigquery:** Reduce default backoffs ([#10558](https://togithub.com/googleapis/google-cloud-go/issues/10558)) ([037e9ef](https://togithub.com/googleapis/google-cloud-go/commit/037e9efa929ad9f8d6f725b28ec8096c3e536b76)) ### Documentation * **bigquery/analyticshub:** A comment for message `DataExchange` is changed ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery/analyticshub:** A comment for message `Listing` is changed ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery/datatransfer:** Update OAuth links in `CreateTransferConfigRequest` and `UpdateTransferConfigRequest` ([3df3c04](https://togithub.com/googleapis/google-cloud-go/commit/3df3c04f0dffad3fa2fe272eb7b2c263801b9ada)) * **bigquery:** Improve Inserter and StructSaver godoc ([#10170](https://togithub.com/googleapis/google-cloud-go/issues/10170)) ([c1cffb6](https://togithub.com/googleapis/google-cloud-go/commit/c1cffb63c33ae49f3a705bd0bc7a32cd2b0319bc)) * **bigquery:** Update description of query preview feature ([#10554](https://togithub.com/googleapis/google-cloud-go/issues/10554)) ([25c5cbe](https://togithub.com/googleapis/google-cloud-go/commit/25c5cbe6f31d62fdea1455889ac2e336d1287615)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
eliben
pushed a commit
to eliben/google-cloud-go
that referenced
this pull request
Jul 23, 2024
🤖 I have created a release *beep* *boop* --- ## [1.62.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.61.0...bigquery/v1.62.0) (2024-07-22) ### Features * **bigquery/analyticshub:** Support Direct Table Access Toggle (Egress GA) ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery/analyticshub:** Support public directory self service for Listings/Exchanges ([googleapis#10485](https://togithub.com/googleapis/google-cloud-go/issues/10485)) ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery:** Add rounding mode to FieldSchema ([googleapis#10328](https://togithub.com/googleapis/google-cloud-go/issues/10328)) ([1a9e204](https://togithub.com/googleapis/google-cloud-go/commit/1a9e204d7752c5bfe9edfd7bc7ee36c5b1385783)) * **bigquery:** Json support on managedwriter/adapt pkg ([googleapis#10542](https://togithub.com/googleapis/google-cloud-go/issues/10542)) ([978d4a1](https://togithub.com/googleapis/google-cloud-go/commit/978d4a1e47cbd8d4bf567b616381a2f12fac4cab)) * **bigquery:** Support column name character map in load jobs ([googleapis#10425](https://togithub.com/googleapis/google-cloud-go/issues/10425)) ([b829327](https://togithub.com/googleapis/google-cloud-go/commit/b82932789af82b5e0799c20a096aab98132b5eb1)) ### Bug Fixes * **bigquery/storage/managedwriter:** Faster context failure on send ([googleapis#10169](https://togithub.com/googleapis/google-cloud-go/issues/10169)) ([1fb0e64](https://togithub.com/googleapis/google-cloud-go/commit/1fb0e6401d584bf8ede60a170b4d82dc211010b8)) * **bigquery:** Bump google.golang.org/api@v0.187.0 ([8fa9e39](https://togithub.com/googleapis/google-cloud-go/commit/8fa9e398e512fd8533fd49060371e61b5725a85b)) * **bigquery:** Bump google.golang.org/grpc@v1.64.1 ([8ecc4e9](https://togithub.com/googleapis/google-cloud-go/commit/8ecc4e9622e5bbe9b90384d5848ab816027226c5)) * **bigquery:** Empty slice instead of nil slice for primitive repeated fields ([googleapis#7315](https://togithub.com/googleapis/google-cloud-go/issues/7315)) ([b371210](https://togithub.com/googleapis/google-cloud-go/commit/b3712100831061fea8605e574d482d7f768ecf14)) * **bigquery:** Reduce default backoffs ([googleapis#10558](https://togithub.com/googleapis/google-cloud-go/issues/10558)) ([037e9ef](https://togithub.com/googleapis/google-cloud-go/commit/037e9efa929ad9f8d6f725b28ec8096c3e536b76)) ### Documentation * **bigquery/analyticshub:** A comment for message `DataExchange` is changed ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery/analyticshub:** A comment for message `Listing` is changed ([b660d68](https://togithub.com/googleapis/google-cloud-go/commit/b660d6870658fe6881883785bcebaea0929fec0a)) * **bigquery/datatransfer:** Update OAuth links in `CreateTransferConfigRequest` and `UpdateTransferConfigRequest` ([3df3c04](https://togithub.com/googleapis/google-cloud-go/commit/3df3c04f0dffad3fa2fe272eb7b2c263801b9ada)) * **bigquery:** Improve Inserter and StructSaver godoc ([googleapis#10170](https://togithub.com/googleapis/google-cloud-go/issues/10170)) ([c1cffb6](https://togithub.com/googleapis/google-cloud-go/commit/c1cffb63c33ae49f3a705bd0bc7a32cd2b0319bc)) * **bigquery:** Update description of query preview feature ([googleapis#10554](https://togithub.com/googleapis/google-cloud-go/issues/10554)) ([25c5cbe](https://togithub.com/googleapis/google-cloud-go/commit/25c5cbe6f31d62fdea1455889ac2e336d1287615)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
This was referenced Jul 30, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(bigquery): reduce the initial backoff for job polling
This PR reduces initial backoffs and multipliers used during job polling. Previously we used a 1s starting backoff, this has been reduced to 50ms. In job execution most time is typically spent in server, waiting for the hanging GetQueryResults response.
This should in generally reduce wasted wait for jobs that poll for multiple iterations.
Fixes: #10555