Comparing changes

…63.0 (#1897) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-sdk/v4](https://togithub.com/cloudquery/plugin-sdk) | require | minor | `v4.62.0` -> `v4.63.0` | --- ### Release Notes <details> <summary>cloudquery/plugin-sdk (github.com/cloudquery/plugin-sdk/v4)</summary> ### [`v4.63.0`](https://togithub.com/cloudquery/plugin-sdk/releases/tag/v4.63.0) [Compare Source](https://togithub.com/cloudquery/plugin-sdk/compare/v4.62.0...v4.63.0) ##### Features - Add sync sharding ([#1891](https://togithub.com/cloudquery/plugin-sdk/issues/1891)) ([e1823f8](https://togithub.com/cloudquery/plugin-sdk/commit/e1823f82fd3c457f1f58c266bfd9519b547f31c9)) ##### Bug Fixes - **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.22.3 ([#1895](https://togithub.com/cloudquery/plugin-sdk/issues/1895)) ([b05d24b](https://togithub.com/cloudquery/plugin-sdk/commit/b05d24b345ec519deef70156377338e6b41d8108)) - **deps:** Update module google.golang.org/grpc to v1.66.2 ([#1893](https://togithub.com/cloudquery/plugin-sdk/issues/1893)) ([6d70b88](https://togithub.com/cloudquery/plugin-sdk/commit/6d70b88808aa144c4c05e007b291bd8d958858e4)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

instead of `WithAccessToken`, which needed explicit default values and didn't support refresh tokens in non-cloud mode. --------- Co-authored-by: Kemal Hadimli <disq@users.noreply.github.com>

…s` for scheduler (#1900) Co-authored-by: Kemal Hadimli <disq@users.noreply.github.com>

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/aws/aws-sdk-go-v2](https://togithub.com/aws/aws-sdk-go-v2) | require | minor | `v1.30.5` -> `v1.31.0` | | [github.com/aws/aws-sdk-go-v2/config](https://togithub.com/aws/aws-sdk-go-v2) | require | patch | `v1.27.33` -> `v1.27.36` | | [github.com/aws/aws-sdk-go-v2/service/licensemanager](https://togithub.com/aws/aws-sdk-go-v2) | require | minor | `v1.27.6` -> `v1.28.0` | | [github.com/aws/aws-sdk-go-v2/service/marketplacemetering](https://togithub.com/aws/aws-sdk-go-v2) | require | minor | `v1.23.6` -> `v1.24.0` | --- ### Release Notes <details> <summary>aws/aws-sdk-go-v2 (github.com/aws/aws-sdk-go-v2)</summary> ### [`v1.31.0`](https://togithub.com/aws/aws-sdk-go-v2/compare/v1.30.5...v1.31.0) [Compare Source](https://togithub.com/aws/aws-sdk-go-v2/compare/v1.30.5...v1.31.0) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Saturday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [google.golang.org/grpc](https://togithub.com/grpc/grpc-go) | require | minor | `v1.66.2` -> `v1.67.0` | --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.67.0`](https://togithub.com/grpc/grpc-go/releases/tag/v1.67.0): Release 1.67.0 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.66.2...v1.67.0) ### Bug Fixes - ringhash: when used with multiple EDS priorities, fix bug that could prevent a higher priority from recovering from transient failure. ([#7364](https://togithub.com/grpc/grpc-go/issues/7364)) - Special Thanks: [@atollena](https://togithub.com/atollena) ### Behavior Changes - In accordance with [RFC 7540](https://tools.ietf.org/html/rfc7540#section-3.3), clients and servers will now reject TLS connections that don't support ALPN. This can be disabled by setting the environment variable `GRPC_ENFORCE_ALPN_ENABLED` to `false` (case insensitive). Please file a bug if you encounter any issues with this behavior. The environment variable to revert this behavior will be removed in an upcoming release. ([#7535](https://togithub.com/grpc/grpc-go/issues/7535)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Saturday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/aws/aws-sdk-go-v2/config](https://togithub.com/aws/aws-sdk-go-v2) | require | patch | `v1.27.36` -> `v1.27.39` | | [github.com/aws/aws-sdk-go-v2/service/licensemanager](https://togithub.com/aws/aws-sdk-go-v2) | require | patch | `v1.28.0` -> `v1.28.3` | | [github.com/aws/aws-sdk-go-v2/service/marketplacemetering](https://togithub.com/aws/aws-sdk-go-v2) | require | patch | `v1.24.0` -> `v1.24.3` | --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Saturday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

## The Problem There's a "bug" in the concurrency formula on the scheduler (all strategies). This is the formula: ```go tableConcurrency := max(s.concurrency/minResourceConcurrency, minTableConcurrency) resourceConcurrency := tableConcurrency * minResourceConcurrency ``` But these values are hardcoded: ```go minResourceConcurrency := 100 minTableConcurrency := 1 ``` So if you replace: ```go tableConcurrency := max(s.concurrency/100, 1) resourceConcurrency := tableConcurrency * 100 ``` This means that any plugin whose default concurrency is `<= 100` will have a table concurrency of `1`, even if it's the only table being synced (assuming no one changes the default). ## The Fix I made a very subtle change in the formula. Only if concurrency is `<= 100`, I change the `minResourceConcurrency` to `concurrency/10`. This decreases the resource concurrency up to 10x (that we don't seem to be hitting anyway), and increases the table concurrency up to 10x. ## Plugins affected (on default concurrency) - bamboo-hr - bigquery - ~clickhouse~ (doesn't use scheduler) - confluence - crowddev - ~file~ (doesn't use scheduler) - leanix - oracledb - ~s3~ (doesn't use scheduler) - sentinelone - servicenow - shopify - sonarqube - statuspage ## The Results I'm still working on the results (it's trickier than it seems). In principle, they are very encouraging: ### BigQuery **Before** ``` $ cli sync bigquery_to_postgresql.yaml Loading spec(s) from bigquery_to_postgresql.yaml Starting sync for: bigquery (cloudquery/bigquery@v1.7.0) -> [postgresql (cloudquery/postgresql@v8.6.0)] Sync completed successfully. Resources: 26139, Errors: 0, Warnings: 0, Time: 2m4s ``` **After** ``` $ cli sync bigquery_to_postgresql.yaml Loading spec(s) from bigquery_to_postgresql.yaml Starting sync for: bigquery (cloudquery/bigquery@v1.7.0) -> [postgresql (cloudquery/postgresql@v8.6.0)] Sync completed successfully. Resources: 26139, Errors: 0, Warnings: 0, Time: 1m27s ``` **Result** 1.43x of regular speed (43% faster) ### Sentinelone **Before** ``` $ cli sync . Loading spec(s) from . Starting sync for: sentinelone (grpc@localhost:7777) -> [postgresql (cloudquery/postgresql@v8.5.5)] Sync completed successfully. Resources: 1231, Errors: 0, Warnings: 0, Time: 1m4s ``` **After** ``` $ cli sync . Loading spec(s) from . Starting sync for: sentinelone (grpc@localhost:7777) -> [postgresql (cloudquery/postgresql@v8.5.5)] Sync completed successfully. Resources: 1231, Errors: 0, Warnings: 0, Time: 15s ``` **Result** 4.27x of regular speed (327% faster) ### Sonarqube **Before** ``` $ cli sync sonarqube_to_postgresql.yaml Loading spec(s) from sonarqube_to_postgresql.yaml Starting sync for: sonarqube (grpc@localhost:7777) -> [postgresql (cloudquery/postgresql@v8.6.0)] Sync completed successfully. Resources: 4594, Errors: 0, Warnings: 0, Time: 39s ``` **After** ``` $ cli sync sonarqube_to_postgresql.yaml Loading spec(s) from sonarqube_to_postgresql.yaml Starting sync for: sonarqube (grpc@localhost:7777) -> [postgresql (cloudquery/postgresql@v8.6.0)] Sync completed successfully. Resources: 4594, Errors: 0, Warnings: 0, Time: 22s ``` **Result** 1.77x of regular speed (77% faster)

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [golangci/golangci-lint](https://togithub.com/golangci/golangci-lint) | minor | `v1.60.3` -> `v1.61.0` | --- ### Release Notes <details> <summary>golangci/golangci-lint (golangci/golangci-lint)</summary> ### [`v1.61.0`](https://togithub.com/golangci/golangci-lint/blob/HEAD/CHANGELOG.md#v1610) [Compare Source](https://togithub.com/golangci/golangci-lint/compare/v1.60.3...v1.61.0) 1. Enhancements - Add `junit-xml-extended` format - Exclude Swagger Codegen files by default 2. Updated linters - `dupword`: from 0.0.14 to 0.1.1 - `fatcontext`: from 0.4.0 to 0.5.2 - `gci`: from 0.13.4 to 0.13.5 (new option `no-lex-order`) - `go-ruleguard`: from 0.4.2 to [`0fe6f58`](https://togithub.com/golangci/golangci-lint/commit/0fe6f58b47b1) (fix panic with custom linters) - `godot`: from 1.4.16 to 1.4.17 - `gomodguard`: from 1.3.3 to 1.3.5 - `gosec`: disable temporarily `G407` - `gosec`: from [`ab3f6c1`](https://togithub.com/golangci/golangci-lint/commit/ab3f6c1c83a0) to 2.21.2 (partially fix `G115`) - `intrange`: from 0.1.2 to 0.2.0 - `nolintlint`: remove the empty line in the directive replacement 3. Misc. - Improve runtime version parsing 4. Documentation - Add additional info about `typecheck` </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | golang.org/x/oauth2 | require | minor | `v0.22.0` -> `v0.23.0` | --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

#1912) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-pb-go](https://togithub.com/cloudquery/plugin-pb-go) | require | patch | `v1.22.3` -> `v1.22.4` | --- ### Release Notes <details> <summary>cloudquery/plugin-pb-go (github.com/cloudquery/plugin-pb-go)</summary> ### [`v1.22.4`](https://togithub.com/cloudquery/plugin-pb-go/releases/tag/v1.22.4) [Compare Source](https://togithub.com/cloudquery/plugin-pb-go/compare/v1.22.3...v1.22.4) ##### Bug Fixes - **deps:** Update module google.golang.org/grpc to v1.67.0 ([#408](https://togithub.com/cloudquery/plugin-pb-go/issues/408)) ([a6167a8](https://togithub.com/cloudquery/plugin-pb-go/commit/a6167a8de7807cb96604070a4328b7ff5109776a)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

🤖 I have created a release *beep* *boop* --- ## [4.64.0](v4.63.0...v4.64.0) (2024-10-01) ### Features * Add `opts.SchedulerOpts()` helper to convert `plugin.SyncOptions` for scheduler ([#1900](#1900)) ([242fb55](242fb55)) * **remoteoauth:** Add `WithToken` option ([#1898](#1898)) ([ff7a485](ff7a485)) * Update concurrency formula. ([#1907](#1907)) ([adce99c](adce99c)) ### Bug Fixes * **deps:** Update aws-sdk-go-v2 monorepo ([#1903](#1903)) ([ce2a0ef](ce2a0ef)) * **deps:** Update aws-sdk-go-v2 monorepo ([#1908](#1908)) ([bea3b00](bea3b00)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.22.4 ([#1912](#1912)) ([c28aabe](c28aabe)) * **deps:** Update module golang.org/x/oauth2 to v0.23.0 ([#1910](#1910)) ([6fe6414](6fe6414)) * **deps:** Update module google.golang.org/grpc to v1.67.0 ([#1904](#1904)) ([a349812](a349812)) * **deps:** Update opentelemetry-go monorepo ([#1911](#1911)) ([78e05e1](78e05e1)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing changes

Open a pull request

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 21, 2024

Commits on Sep 28, 2024

Commits on Sep 30, 2024

Commits on Oct 1, 2024

Commits on Oct 2, 2024

This comparison is taking too long to generate.

Uh oh!