Comparing changes

…16.1 (#1324) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-sdk/v4](https://togithub.com/cloudquery/plugin-sdk) | require | patch | `v4.16.0` -> `v4.16.1` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>cloudquery/plugin-sdk (github.com/cloudquery/plugin-sdk/v4)</summary> ### [`v4.16.1`](https://togithub.com/cloudquery/plugin-sdk/releases/tag/v4.16.1) [Compare Source](https://togithub.com/cloudquery/plugin-sdk/compare/v4.16.0...v4.16.1) ##### Bug Fixes - **package:** Only return one level down of relations when writing `tables.json` ([#1321](https://togithub.com/cloudquery/plugin-sdk/issues/1321)) ([3d4ebe0](https://togithub.com/cloudquery/plugin-sdk/commit/3d4ebe0098ba4e458d88e092e6240ee848c38c0a)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

This adds an optional `OnBeforeSend` hook to plugin clients that choose to implement the interface. When implemented, the plugin client will receive a call before every message is sent, allowing it to modify the message, perform a side-effect, or return an error. This keeps the potentially general useful for all plugins, with the first use case being to check whether a row should be counted as premium or not in upcoming paid plugins.

This adds an `IsPaid` flag to the table definition. This can be used by premium plugins to indicate that a certain table is paid.

#1329) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/arrow/go/v14](https://togithub.com/cloudquery/arrow) | replace | digest | `dbcb149` -> `f46436f` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

) This will allows a client to update the usage asynchronously. The updater will only call the API when a configurable number of rows have been updated or a timeout is reached. In addition the updater will flush any remaining rows before closing. If an error is encountered then an exponential backoff up to a max number of retries or a maximum wait time (relative to the start time of the last query) will be followed. In addition, if the server replies with a status of `429` and includes a `Retry-After` header, then the client will wait that number of seconds before retrying. It might be worth considering just using a retry library also e.g. https://github.com/avast/retry-go.

…#1331) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [google.golang.org/grpc](https://togithub.com/grpc/grpc-go) | require | patch | `v1.58.2` -> `v1.58.3` | | [google.golang.org/grpc](https://togithub.com/grpc/grpc-go) | indirect | patch | `v1.58.2` -> `v1.58.3` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. ### GitHub Vulnerability Alerts #### [GHSA-m425-mq94-257g](https://togithub.com/grpc/grpc-go/security/advisories/GHSA-m425-mq94-257g) ### Impact In affected releases of gRPC-Go, it is possible for an attacker to send HTTP/2 requests, cancel them, and send subsequent requests, which is valid by the HTTP/2 protocol, but would cause the gRPC-Go server to launch more concurrent method handlers than the configured maximum stream limit. ### Patches This vulnerability was addressed by #6703 and has been included in patch releases: 1.56.3, 1.57.1, 1.58.3. It is also included in the latest release, 1.59.0. Along with applying the patch, users should also ensure they are using the `grpc.MaxConcurrentStreams` server option to apply a limit to the server's resources used for any single connection. ### Workarounds None. ### References #6703 --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.58.3`](https://togithub.com/grpc/grpc-go/releases/tag/v1.58.3): Release 1.58.3 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.58.2...v1.58.3) ##### Security - server: prohibit more than MaxConcurrentStreams handlers from running at once (CVE-2023-44487) In addition to this change, applications should ensure they do not leave running tasks behind related to the RPC before returning from method handlers, or should enforce appropriate limits on any such work. </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

#1332) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-pb-go](https://togithub.com/cloudquery/plugin-pb-go) | require | minor | `v1.12.3` -> `v1.13.0` | | [github.com/cloudquery/plugin-pb-go](https://togithub.com/cloudquery/plugin-pb-go) | indirect | minor | `v1.12.3` -> `v1.13.0` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>cloudquery/plugin-pb-go (github.com/cloudquery/plugin-pb-go)</summary> ### [`v1.13.0`](https://togithub.com/cloudquery/plugin-pb-go/releases/tag/v1.13.0) [Compare Source](https://togithub.com/cloudquery/plugin-pb-go/compare/v1.12.3...v1.13.0) ##### Features - Allow passing an auth token when downloading Hub plugins ([#138](https://togithub.com/cloudquery/plugin-pb-go/issues/138)) ([b9f491d](https://togithub.com/cloudquery/plugin-pb-go/commit/b9f491d049329437c20ae140eb02b1cb3a3add95)) ##### Bug Fixes - **deps:** Update github.com/cloudquery/arrow/go/v14 digest to [`dbcb149`](https://togithub.com/cloudquery/plugin-pb-go/commit/dbcb149) ([#130](https://togithub.com/cloudquery/plugin-pb-go/issues/130)) ([e91be81](https://togithub.com/cloudquery/plugin-pb-go/commit/e91be81c62b799d67810550e70dfa5c1cf253c4e)) - **deps:** Update github.com/cloudquery/arrow/go/v14 digest to [`f46436f`](https://togithub.com/cloudquery/plugin-pb-go/commit/f46436f) ([#137](https://togithub.com/cloudquery/plugin-pb-go/issues/137)) ([f92d39c](https://togithub.com/cloudquery/plugin-pb-go/commit/f92d39c67536f41473abef7d6d9ebc30015c393e)) - **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.2.9 ([#133](https://togithub.com/cloudquery/plugin-pb-go/issues/133)) ([866eb55](https://togithub.com/cloudquery/plugin-pb-go/commit/866eb557e8cdbe593d44a30449d2c8ea9defb401)) - **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.3.0 ([#134](https://togithub.com/cloudquery/plugin-pb-go/issues/134)) ([5ba2f68](https://togithub.com/cloudquery/plugin-pb-go/commit/5ba2f6844b6f446d12ea1978a5f9e941f59eb3cf)) - **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.4.0 ([#135](https://togithub.com/cloudquery/plugin-pb-go/issues/135)) ([9d8001d](https://togithub.com/cloudquery/plugin-pb-go/commit/9d8001ded534d559132e7dd71798d4bbf705df37)) - **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.4.1 ([#136](https://togithub.com/cloudquery/plugin-pb-go/issues/136)) ([6e1bef2](https://togithub.com/cloudquery/plugin-pb-go/commit/6e1bef250fa3b39ce01558ba54aa0f1b7dfb6400)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

…1.4.2 (#1335) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/cloudquery-api-go](https://togithub.com/cloudquery/cloudquery-api-go) | require | patch | `v1.4.1` -> `v1.4.2` | | [github.com/cloudquery/cloudquery-api-go](https://togithub.com/cloudquery/cloudquery-api-go) | indirect | patch | `v1.4.1` -> `v1.4.2` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>cloudquery/cloudquery-api-go (github.com/cloudquery/cloudquery-api-go)</summary> ### [`v1.4.2`](https://togithub.com/cloudquery/cloudquery-api-go/releases/tag/v1.4.2) [Compare Source](https://togithub.com/cloudquery/cloudquery-api-go/compare/v1.4.1...v1.4.2) ##### Bug Fixes - Generate CloudQuery Go API Client from `spec.json` ([#46](https://togithub.com/cloudquery/cloudquery-api-go/issues/46)) ([5f83a82](https://togithub.com/cloudquery/cloudquery-api-go/commit/5f83a82d4cbf79271609c12c612ba495c6613ef6)) - Generate CloudQuery Go API Client from `spec.json` ([#48](https://togithub.com/cloudquery/cloudquery-api-go/issues/48)) ([7dfeae5](https://togithub.com/cloudquery/cloudquery-api-go/commit/7dfeae55de35be96272a16b56cce36b4250e03b0)) - Generate CloudQuery Go API Client from `spec.json` ([#49](https://togithub.com/cloudquery/cloudquery-api-go/issues/49)) ([b67368e](https://togithub.com/cloudquery/cloudquery-api-go/commit/b67368efe71d56298d64b101c92e3c14733cdf0b)) - Generate CloudQuery Go API Client from `spec.json` ([#50](https://togithub.com/cloudquery/cloudquery-api-go/issues/50)) ([eeb9744](https://togithub.com/cloudquery/cloudquery-api-go/commit/eeb9744efd103f9f499355fd68735d8d4d2c6f6d)) - Generate CloudQuery Go API Client from `spec.json` ([#51](https://togithub.com/cloudquery/cloudquery-api-go/issues/51)) ([c7fa339](https://togithub.com/cloudquery/cloudquery-api-go/commit/c7fa339728fa234a9b4923567e2fa42d39f1e7ab)) - Generate CloudQuery Go API Client from `spec.json` ([#52](https://togithub.com/cloudquery/cloudquery-api-go/issues/52)) ([71eb1d8](https://togithub.com/cloudquery/cloudquery-api-go/commit/71eb1d8cc41e859aede30765c7fae3b03d50185d)) - Generate CloudQuery Go API Client from `spec.json` ([#53](https://togithub.com/cloudquery/cloudquery-api-go/issues/53)) ([a67bcba](https://togithub.com/cloudquery/cloudquery-api-go/commit/a67bcba14f1d84b6898cf0d3bacb2bef7e69942d)) - Generate CloudQuery Go API Client from `spec.json` ([#54](https://togithub.com/cloudquery/cloudquery-api-go/issues/54)) ([8b970be](https://togithub.com/cloudquery/cloudquery-api-go/commit/8b970bec590c9cf36c80ece0e8e40629664cf6e9)) - Generate CloudQuery Go API Client from `spec.json` ([#55](https://togithub.com/cloudquery/cloudquery-api-go/issues/55)) ([0c91565](https://togithub.com/cloudquery/cloudquery-api-go/commit/0c915656da7e6cff881848bd1295896805c5e205)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

#1336) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-pb-go](https://togithub.com/cloudquery/plugin-pb-go) | require | patch | `v1.13.0` -> `v1.13.1` | | [github.com/cloudquery/plugin-pb-go](https://togithub.com/cloudquery/plugin-pb-go) | indirect | patch | `v1.13.0` -> `v1.13.1` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>cloudquery/plugin-pb-go (github.com/cloudquery/plugin-pb-go)</summary> ### [`v1.13.1`](https://togithub.com/cloudquery/plugin-pb-go/releases/tag/v1.13.1) [Compare Source](https://togithub.com/cloudquery/plugin-pb-go/compare/v1.13.0...v1.13.1) ##### Bug Fixes - **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.4.2 ([#139](https://togithub.com/cloudquery/plugin-pb-go/issues/139)) ([b84ffc9](https://togithub.com/cloudquery/plugin-pb-go/commit/b84ffc9c26c43e4ca4e64484676bf6a17bdc7f54)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

#1337) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/arrow/go/v14](https://togithub.com/cloudquery/arrow) | replace | digest | `f46436f` -> `50d3871` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information. --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

In some cases, when the resource channel is processing a large collection of objects, the sync process can continue even if the context cancel has been triggered. This is problematic for billing and usage since a sync can significantly overshoot the remaining rows quota. To allow an in-progress sync to abort quicker, we need to modify a number of channel patterns from using `range` to using `select` - which will allow the `context.Done` channel to be monitored. This PR also adds the `OnSyncFinisher` hook which is called when the sync has finished adding messages. This allows the client code to call the batch updater close method without missing any updates. fixes: cloudquery/cloudquery-issues#750

This adds a background monitoring process that will periodically check the remaining quota for a premium plugin. If the quota is exceeded then a context cancellation is triggered, forcing the sync process to stop. fixes: cloudquery/cloudquery-issues#749

🤖 I have created a release *beep* *boop* --- ## [4.17.0](v4.16.1...v4.17.0) (2023-10-30) ### Features * Add IsPaid flag to table definition ([#1327](#1327)) ([ffd14bf](ffd14bf)) * Add OnBeforeSend hook ([#1325](#1325)) ([023ebbc](023ebbc)) * Adding a batch updater to allow usage updates to be batched ([#1326](#1326)) ([0301ed7](0301ed7)) * Adding quota monitoring for premium plugins ([#1333](#1333)) ([b7a2ca5](b7a2ca5)) * Allow sync to be cancelled when in progress ([#1334](#1334)) ([6d7be0b](6d7be0b)) ### Bug Fixes * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to 50d3871 ([#1337](#1337)) ([f15a89d](f15a89d)) * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to f46436f ([#1329](#1329)) ([ee24384](ee24384)) * **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.4.2 ([#1335](#1335)) ([2ecd2a1](2ecd2a1)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.13.0 ([#1332](#1332)) ([5553f85](5553f85)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.13.1 ([#1336](#1336)) ([b782ee7](b782ee7)) * **deps:** Update module google.golang.org/grpc to v1.58.3 [SECURITY] ([#1331](#1331)) ([43f60c2](43f60c2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing changes

Open a pull request

Commits on Oct 19, 2023

Commits on Oct 20, 2023

Commits on Oct 23, 2023

Commits on Oct 24, 2023

Commits on Oct 25, 2023

Commits on Oct 26, 2023

Commits on Oct 27, 2023

Commits on Oct 30, 2023

This comparison is taking too long to generate.

Uh oh!