Comparing changes

Updates the CLI latest version to v6.6.3

…#19126) #### Summary This PR adds a new transformation type that allows users to add a column whose value is the timestamp at the point in time that the transformer handles the data

🤖 I have created a release *beep* *boop* --- ## [1.1.0](plugins-transformer-basic-v1.0.2...plugins-transformer-basic-v1.1.0) (2024-09-13) ### Features * Add new transform that adds a column with the current timestamp ([#19126](#19126)) ([78d8065](78d8065)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

….0 (#19137) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-sdk/v4](https://togithub.com/cloudquery/plugin-sdk) | require | minor | `v4.61.0` -> `v4.62.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>cloudquery/plugin-sdk (github.com/cloudquery/plugin-sdk/v4)</summary> ### [`v4.62.0`](https://togithub.com/cloudquery/plugin-sdk/releases/tag/v4.62.0) [Compare Source](https://togithub.com/cloudquery/plugin-sdk/compare/v4.61.0...v4.62.0) ##### Features - Support Contract Listing For AWS Marketplace ([#1889](https://togithub.com/cloudquery/plugin-sdk/issues/1889)) ([4654866](https://togithub.com/cloudquery/plugin-sdk/commit/4654866cb423d237cddb696384e910f59539e1d9)) ##### Bug Fixes - **deps:** Update aws-sdk-go-v2 monorepo ([#1890](https://togithub.com/cloudquery/plugin-sdk/issues/1890)) ([b185e11](https://togithub.com/cloudquery/plugin-sdk/commit/b185e11bad937fbbeb9178f88f0ede749088efc7)) - **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.22.2 ([#1887](https://togithub.com/cloudquery/plugin-sdk/issues/1887)) ([a881fac](https://togithub.com/cloudquery/plugin-sdk/commit/a881fac8976ecfb83101c6268d114e28e19bd2f2)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [google.golang.org/grpc](https://togithub.com/grpc/grpc-go) | require | patch | `v1.66.0` -> `v1.66.2` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.66.2`](https://togithub.com/grpc/grpc-go/releases/tag/v1.66.2): Release 1.66.2 [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.66.1...v1.66.2) ### Dependencies - Remove unintentional dependency on the `testing` package ([#7579](https://togithub.com/grpc/grpc-go/issues/7579)) - Remove unintentional dependency on the `flate` package ([#7595](https://togithub.com/grpc/grpc-go/issues/7595)) - Special Thanks: [@ash2k](https://togithub.com/ash2k) ### Bug Fixes - client: fix a bug that prevented memory reuse after handling unary RPCs ([#7571](https://togithub.com/grpc/grpc-go/issues/7571)) - Special Thanks: [@coxley](https://togithub.com/coxley) ### [`v1.66.1`](https://togithub.com/grpc/grpc-go/compare/v1.66.0...v1.66.1) [Compare Source](https://togithub.com/grpc/grpc-go/compare/v1.66.0...v1.66.1) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Saturday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [google.golang.org/api](https://togithub.com/googleapis/google-api-go-client) | require | minor | `v0.196.0` -> `v0.197.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>googleapis/google-api-go-client (google.golang.org/api)</summary> ### [`v0.197.0`](https://togithub.com/googleapis/google-api-go-client/releases/tag/v0.197.0) [Compare Source](https://togithub.com/googleapis/google-api-go-client/compare/v0.196.0...v0.197.0) ##### Features - **all:** Auto-regenerate discovery clients ([#2770](https://togithub.com/googleapis/google-api-go-client/issues/2770)) ([07782e2](https://togithub.com/googleapis/google-api-go-client/commit/07782e2db0dc8432b659fb6b1392837a9a33d784)) - **all:** Auto-regenerate discovery clients ([#2772](https://togithub.com/googleapis/google-api-go-client/issues/2772)) ([0c7c227](https://togithub.com/googleapis/google-api-go-client/commit/0c7c227b6d00a2f22de990156f2f75a3862f77a7)) - **all:** Auto-regenerate discovery clients ([#2774](https://togithub.com/googleapis/google-api-go-client/issues/2774)) ([6476ddd](https://togithub.com/googleapis/google-api-go-client/commit/6476dddd4810be9b17e5692f6950f3d24d52abe7)) - **all:** Auto-regenerate discovery clients ([#2775](https://togithub.com/googleapis/google-api-go-client/issues/2775)) ([11e2ff6](https://togithub.com/googleapis/google-api-go-client/commit/11e2ff6ea82ce403f464ea42cf23ec702b6ef883)) - **all:** Auto-regenerate discovery clients ([#2776](https://togithub.com/googleapis/google-api-go-client/issues/2776)) ([cc62887](https://togithub.com/googleapis/google-api-go-client/commit/cc6288771612f80e5d2338dcf5897b7acd0952dd)) - **all:** Auto-regenerate discovery clients ([#2778](https://togithub.com/googleapis/google-api-go-client/issues/2778)) ([72cc5c4](https://togithub.com/googleapis/google-api-go-client/commit/72cc5c4f6ef206c857f222653dff96907b320788)) - **all:** Auto-regenerate discovery clients ([#2779](https://togithub.com/googleapis/google-api-go-client/issues/2779)) ([cedc5b0](https://togithub.com/googleapis/google-api-go-client/commit/cedc5b01c4999c17c97f4f6336fb2f2cae318453)) ##### Bug Fixes - **transport:** Set UniverseDomain in http.NewClient for new auth ([#2773](https://togithub.com/googleapis/google-api-go-client/issues/2773)) ([140d0a5](https://togithub.com/googleapis/google-api-go-client/commit/140d0a5e70ebd3f12a6ec11ec48880e6d86cf2bc)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Saturday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

#### Summary Fixes #19122 Before, the `INSERT` query was built within the `appendTableBatch` and `overwriteTableBatch` functions. But we aren't yet aware of how many rows need to be inserted, so we don't know how many value blocks to add. Rows are iterated within `writeResources` to create the final records to insert, so this is an opportunity to calculate a row count. So I moved the query building to within `writeResources` and constructed the `INSERT` query based on the amount of rows to be inserted. This reduced the time to write 27,000 records from 20 minutes to under 1 minute. --------- Co-authored-by: erezrokah <erezrokah@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [5.2.8](plugins-destination-mysql-v5.2.7...plugins-destination-mysql-v5.2.8) (2024-09-16) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.62.0 ([#19137](#19137)) ([ed315d0](ed315d0)) * **main:** Write to mysql in bulk ([#19123](#19123)) ([0d1b9d5](0d1b9d5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

@disq

fixes cloudquery/cloudquery-issues#2436 Several different users have reported that, since transformers have been implemented, certain syncs are hanging forever. The transformations implementation seems to work well in these cases: - A sync completes with no errors - A sync finishes with errors due to the source - A sync finishes with errors due to the transformation However, when a destination has an error, at least sometimes it hangs forever. Each transformer is an implementation of a `grpc.BidiStreamingClient`, which uses these two methods: ```go Send(*Req) error Recv() (*Res, error) ``` Which in turn call these: ```go // SendMsg is generally called by generated code. On error, SendMsg aborts // the stream. If the error was generated by the client, the status is // returned directly; otherwise, io.EOF is returned and the status of // the stream may be discovered using RecvMsg. // // SendMsg blocks until: // - There is sufficient flow control to schedule m with the transport, or // - The stream is done, or // - The stream breaks. // // SendMsg does not wait until the message is received by the server. An // untimely stream closure may result in lost messages. To ensure delivery, // users should ensure the RPC completed successfully using RecvMsg. // // It is safe to have a goroutine calling SendMsg and another goroutine // calling RecvMsg on the same stream at the same time, but it is not safe // to call SendMsg on the same stream in different goroutines. It is also // not safe to call CloseSend concurrently with SendMsg. // // It is not safe to modify the message after calling SendMsg. Tracing // libraries and stats handlers may use the message lazily. SendMsg(m any) error // RecvMsg blocks until it receives a message into m or the stream is // done. It returns io.EOF when the stream completes successfully. On // any other error, the stream is aborted and the error contains the RPC // status. // // It is safe to have a goroutine calling SendMsg and another goroutine // calling RecvMsg on the same stream at the same time, but it is not // safe to call RecvMsg on the same stream in different goroutines. RecvMsg(m any) error ``` Both `SendMsg` and `RecvMsg` can hang forever, and this is what users are reporting. However we may want to mitigate this issue (let's say the flow of data or the orchestration for closing is wrong), it's probably gonna be fragile if we still rely on those methods not hanging. Thus, what I've done is running those two methods (`Recv()` and `Send()`) asynchronously. The Pipeline thread will invoke these, but it will exit if someone closes the pipeline (by using a `select`). I've tested running @disq 's reproduction test 100 times; it no longer hangs nor panics: #19130 (comment) **It correctly fails on a test error on insert configuration** ![Screenshot 2024-09-16 at 12 46 59](https://github.com/user-attachments/assets/6147c351-be56-4dc0-bc71-04d81f12db82) **It correctly succeeds on a success test configuration** ![Screenshot 2024-09-16 at 12 47 29](https://github.com/user-attachments/assets/5e5f0259-a972-49ff-8446-e0ed888de739) **It still doesn't swallow the last batch, as tested by running a stable sync multiple times and seeing the same resource count** ![Screenshot 2024-09-16 at 12 57 17](https://github.com/user-attachments/assets/73f65f27-1896-4333-8ba9-19d8d3ff5833)

#### Summary This PR allows for transformer plugins to be downloaded by the `cloudquery install` command

🤖 I have created a release *beep* *boop* --- ## [6.7.0](cli-v6.6.3...cli-v6.7.0) (2024-09-16) ### Features * Add `install` support for Transformer plugins ([#19166](#19166)) ([cc6ca4b](cc6ca4b)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.62.0 ([#19137](#19137)) ([ed315d0](ed315d0)) * **deps:** Update module google.golang.org/grpc to v1.66.2 ([#19160](#19160)) ([8699db3](8699db3)) * Fix transformers-related hang. ([#19165](#19165)) ([e151df2](e151df2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing changes

Open a pull request

Commits on Sep 13, 2024

Commits on Sep 14, 2024

Commits on Sep 16, 2024

This comparison is taking too long to generate.

Uh oh!