feat: Adding a batch updater to allow usage updates to be batched#1326
Merged
kodiakhq[bot] merged 14 commits intomainfrom Oct 24, 2023
Merged
feat: Adding a batch updater to allow usage updates to be batched#1326kodiakhq[bot] merged 14 commits intomainfrom
kodiakhq[bot] merged 14 commits intomainfrom
Conversation
ad97e41 to
0596175
Compare
This will allows a client to update the usage asynchronously. The updater will only call the API when a configurable number of rows have been updated or a timeout is reached. In addition the updater will flush any remaining rows before closing.
4cf66fb to
d4b85a3
Compare
c26cb27 to
22de9a3
Compare
85c4412 to
4484abd
Compare
…o allow a minimum update interval to be set also
4484abd to
c2f5be4
Compare
Contributor
hermanschaaf
left a comment
There was a problem hiding this comment.
Looks great! Left a couple of comments, nothing major
premium/usage.go
Outdated
| defaultMaxRetries = 5 | ||
| defaultMaxWaitTime = 60 * time.Second | ||
| defaultMinimumUpdateDuration = 10 * time.Second | ||
| defaultFlushDuration = 30 * time.Second |
Contributor
There was a problem hiding this comment.
Suggested change
| defaultFlushDuration = 30 * time.Second | |
| defaultMaxTimeBetweenFlushes = 30 * time.Second |
(nit)
premium/usage.go
Outdated
| defaultBatchLimit = 1000 | ||
| defaultMaxRetries = 5 | ||
| defaultMaxWaitTime = 60 * time.Second | ||
| defaultMinimumUpdateDuration = 10 * time.Second |
Contributor
There was a problem hiding this comment.
Suggested change
| defaultMinimumUpdateDuration = 10 * time.Second | |
| defaultMinTimeBetweenFlushes = 10 * time.Second |
(nit)
And we'd update the With... functions accordingly
premium/usage.go
Outdated
|
|
||
| // calculateRetryDuration calculates the duration to sleep relative to the query start time before retrying an update | ||
| func (u *BatchUpdater) calculateRetryDuration(statusCode int, headers http.Header, queryStartTime time.Time, retry int) (time.Duration, error) { | ||
| if statusCode == http.StatusTooManyRequests { |
Contributor
There was a problem hiding this comment.
429 is typically for when the client has made too many requests. We can keep handling this, but the server will typically return 503 if it's getting overloaded, so we should handle that as well https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503
Contributor
Author
There was a problem hiding this comment.
We should add 503 as a possible return here
In this case we want to respect the wait time the server has suggested, so just wait.
hermanschaaf
approved these changes
Oct 24, 2023
hermanschaaf
pushed a commit
that referenced
this pull request
Oct 30, 2023
🤖 I have created a release *beep* *boop* --- ## [4.17.0](v4.16.1...v4.17.0) (2023-10-30) ### Features * Add IsPaid flag to table definition ([#1327](#1327)) ([ffd14bf](ffd14bf)) * Add OnBeforeSend hook ([#1325](#1325)) ([023ebbc](023ebbc)) * Adding a batch updater to allow usage updates to be batched ([#1326](#1326)) ([0301ed7](0301ed7)) * Adding quota monitoring for premium plugins ([#1333](#1333)) ([b7a2ca5](b7a2ca5)) * Allow sync to be cancelled when in progress ([#1334](#1334)) ([6d7be0b](6d7be0b)) ### Bug Fixes * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to 50d3871 ([#1337](#1337)) ([f15a89d](f15a89d)) * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to f46436f ([#1329](#1329)) ([ee24384](ee24384)) * **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.4.2 ([#1335](#1335)) ([2ecd2a1](2ecd2a1)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.13.0 ([#1332](#1332)) ([5553f85](5553f85)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.13.1 ([#1336](#1336)) ([b782ee7](b782ee7)) * **deps:** Update module google.golang.org/grpc to v1.58.3 [SECURITY] ([#1331](#1331)) ([43f60c2](43f60c2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will allows a client to update the usage asynchronously. The updater will only call the API when a configurable number of rows have been updated or a timeout is reached. In addition the updater will flush any remaining rows before closing.
If an error is encountered then an exponential backoff up to a max number of retries or a maximum wait time (relative to the start time of the last query) will be followed.
In addition, if the server replies with a status of
429and includes aRetry-Afterheader, then the client will wait that number of seconds before retrying.It might be worth considering just using a retry library also e.g. https://github.com/avast/retry-go.