feat: Migrate cli, plugins and destinations to new type system by yevgenypats · Pull Request #3323 · cloudquery/cloudquery

yevgenypats · 2022-10-30T17:12:02Z

This is instead of #3176

SDK PRs:

cloudquery/plugin-sdk#318
cloudquery/plugin-sdk#320

Previous related CloudQuery PRs:

hermanschaaf · 2022-10-31T10:49:11Z

cli/cmd/sync2.go

+			return fmt.Errorf("failed to initialize destination plugin client for %s: %w", destinationSpec.Name, err)
+		}
+
+		if err := destClients[i].Migrate(ctx, tables); err != nil {


@yevgenypats Maybe now is a good opportunity to change this so that we only migrate the tables specified in the config? This would also be nice for csv, since only the files for tables specified in the config will get created.

Agree. I think they are quite a few improvements that we can/need to do. for example also to spawn only one process per unique destination. but I suggest doing this in follow-up smaller PRs. should be also easier to review and easier/faster for me to develop

hermanschaaf · 2022-10-31T10:53:15Z

cli/cmd/sync2.go

+		return fmt.Errorf("failed to get metrics for source %s: %w", sourceSpec.Name, err)
+	}
+
+	fmt.Printf("Sync completed successfully. Resources: %d, Errors: %d, Panics: %d\n", metrics.TotalResources(), metrics.TotalErrors(), metrics.TotalPanics())


Looks like we're missing the time taken now, which I think was quite nice to have.

hermanschaaf

Leaving a few comments in the meantime, but I haven't quite read through it all yet; looks like there are still some moving parts here so I'll focus more on cloudquery/plugin-sdk#320

plugins/destination/csv/README.md

plugins/destination/csv/client/client_test.go

plugins/destination/postgresql/Makefile

plugins/destination/postgresql/client/client.go

plugins/destination/postgresql/client/client_test.go

erezrokah

Added a few comments, also we need to add the CSV plugin to:

cloudquery/release-please-config.json

Line 22 in 9d59599

"plugins/destination/test": {

cloudquery/.github/workflows/wait_for_required_workflows.yml

Line 51 in 9d59599

"plugins/destination/test",

cloudquery/.github/pr_labeler.yml

Line 23 in 9d59599

postgresql:

May I suggest to put the new CSV plugin in a separate PR?

Also my main issue is that cqtypes is leaking to the resolvers (we had this before actually when resolvers access the table data).
Not sure how to go about it, but maybe we can always access the struct item instead?

plugins/destination/postgresql/Makefile

plugins/destination/postgresql/CHANGELOG.md

plugins/destination/postgresql/client/migrate_test.go.backup

plugins/destination/postgresql/client/types.go

plugins/destination/postgresql/client/write_test.go

plugins/source/aws/resources/services/iam/users_fetch.go

plugins/source/aws/resources/services/lambda/functions_fetch.go

plugins/source/aws/resources/services/s3/buckets_fetch.go

plugins/source/digitalocean/codegen/main.go

plugins/destination/postgresql/CHANGELOG.md

plugins/destination/postgresql/client/migrate.go

plugins/source/digitalocean/docs/tables/digitalocean_domain_records.md

plugins/source/azure/docs/tables/azure_cdn_custom_domains.md

🤖 I have created a release *beep* *boop* --- ## [1.5.0](plugins-destination-postgresql-v1.4.0...plugins-destination-postgresql-v1.5.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for postgresql to v0.13.18 ([#3417](#3417)) ([5c1cfcd](5c1cfcd)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [1.2.0](plugins-source-azure-v1.1.0...plugins-source-azure-v1.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update golang.org/x/exp digest to c99f073 ([#3372](#3372)) ([c64bc54](c64bc54)) * **deps:** Update module github.com/Azure/azure-sdk-for-go/sdk/azcore to v1.1.4 ([#3376](#3376)) ([294708e](294708e)) * **deps:** Update plugin-sdk for azure to v0.13.17 ([#3400](#3400)) ([a244d6c](a244d6c)) * **deps:** Update plugin-sdk for azure to v0.13.18 ([#3410](#3410)) ([36ea67c](36ea67c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [2.3.0](plugins-source-k8s-v2.2.0...plugins-source-k8s-v2.3.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update kubernetes packages to v0.25.3 ([#3375](#3375)) ([8401b8e](8401b8e)) * **deps:** Update plugin-sdk for k8s to v0.13.18 ([#3407](#3407)) ([e6d116d](e6d116d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [1.2.0](plugins-source-terraform-v1.1.0...plugins-source-terraform-v1.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update module github.com/aws/aws-sdk-go to v1.44.127 ([#3377](#3377)) ([b4f5d32](b4f5d32)) * **deps:** Update plugin-sdk for terraform to v0.13.18 ([#3418](#3418)) ([4aa1202](4aa1202)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

This is instead of #3176 SDK PRs: cloudquery/plugin-sdk#318 cloudquery/plugin-sdk#320 Previous related CloudQuery PRs: #3286 Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com> Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [1.5.0](plugins-destination-postgresql-v1.4.0...plugins-destination-postgresql-v1.5.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for postgresql to v0.13.18 ([#3417](#3417)) ([5c1cfcd](5c1cfcd)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [1.2.0](plugins-source-okta-v1.1.0...plugins-source-okta-v1.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for okta to v0.13.18 ([#3416](#3416)) ([1def393](1def393)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [2.2.0](plugins-source-cloudflare-v2.1.0...plugins-source-cloudflare-v2.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for cloudflare to v0.13.17 ([#3402](#3402)) ([0e26848](0e26848)) * **deps:** Update plugin-sdk for cloudflare to v0.13.18 ([#3411](#3411)) ([5e8ce04](5e8ce04)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [1.2.0](plugins-source-github-v1.1.0...plugins-source-github-v1.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for github to v0.13.18 ([#3405](#3405)) ([b6be94c](b6be94c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [2.4.0](plugins-source-gcp-v2.3.0...plugins-source-gcp-v2.4.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for gcp to v0.13.17 ([#3404](#3404)) ([a919a84](a919a84)) * **deps:** Update plugin-sdk for gcp to v0.13.18 ([#3413](#3413)) ([c59bc48](c59bc48)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [2.2.0](plugins-source-heroku-v2.1.0...plugins-source-heroku-v2.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for heroku to v0.13.17 ([#3406](#3406)) ([900439b](900439b)) * **deps:** Update plugin-sdk for heroku to v0.13.18 ([#3415](#3415)) ([fbe8a77](fbe8a77)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [2.2.0](plugins-source-digitalocean-v2.1.0...plugins-source-digitalocean-v2.2.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) ### Bug Fixes * **deps:** Update plugin-sdk for digitalocean to v0.13.17 ([#3403](#3403)) ([e2f2255](e2f2255)) * **deps:** Update plugin-sdk for digitalocean to v0.13.18 ([#3412](#3412)) ([9d19db1](9d19db1)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [4.0.0](plugins-source-aws-v3.8.0...plugins-source-aws-v4.0.0) (2022-11-02) ### ⚠ BREAKING CHANGES * **aws:** Unified tag structure (#3330) ### Features * Add eventbridge resources ([#3160](#3160)) ([67d3a35](67d3a35)) * **aws:** Fraud Detector support ([#3076](#3076)) ([f0e309a](f0e309a)) * Migrate cli, plugins and destinations to new type system ([#3323](#3323)) ([f265a94](f265a94)) * Update AWS Services (new fields) ([#3324](#3324)) ([0b65803](0b65803)) ### Bug Fixes * Add id to aws_cloudfront_cache_policies, region to aws_ec2_transit_gateways ([#3444](#3444)) ([41362e2](41362e2)) * **aws:** Unified tag structure ([#3330](#3330)) ([c9c1e4c](c9c1e4c)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/credentials to v1.12.23 ([#3378](#3378)) ([c33bf73](c33bf73)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/feature/s3/manager to v1.11.37 ([#3379](#3379)) ([3f1a71d](3f1a71d)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/acm to v1.15.2 ([#3380](#3380)) ([b4329ee](b4329ee)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/apigateway to v1.15.22 ([#3381](#3381)) ([5816480](5816480)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/apigatewayv2 to v1.12.20 ([#3385](#3385)) ([2dd39a6](2dd39a6)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/applicationautoscaling to v1.15.20 ([#3386](#3386)) ([0702717](0702717)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/appsync to v1.15.12 ([#3387](#3387)) ([6bb936a](6bb936a)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/athena to v1.18.12 ([#3388](#3388)) ([e95f748](e95f748)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/autoscaling to v1.23.18 ([#3389](#3389)) ([5af977f](5af977f)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/backup to v1.17.11 ([#3390](#3390)) ([1ecaab7](1ecaab7)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cloudfront to v1.20.7 ([#3391](#3391)) ([46465d1](46465d1)) * **deps:** Update plugin-sdk for aws to v0.13.17 ([#3399](#3399)) ([f2cd266](f2cd266)) * **deps:** Update plugin-sdk for aws to v0.13.18 ([#3409](#3409)) ([92fa576](92fa576)) * Filtering for DocDB + Neptune ([#3271](#3271)) ([8080c6e](8080c6e)) * Update endpoints ([#3368](#3368)) ([b78c6b1](b78c6b1)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [2.3.0](cloudquery/cloudquery@plugins-source-k8s-v2.2.0...plugins-source-k8s-v2.3.0) (2022-11-01) ### Features * Migrate cli, plugins and destinations to new type system ([#3323](cloudquery/cloudquery#3323)) ([e51e35f](cloudquery/cloudquery@e51e35f)) ### Bug Fixes * **deps:** Update kubernetes packages to v0.25.3 ([#3375](cloudquery/cloudquery#3375)) ([d9f0feb](cloudquery/cloudquery@d9f0feb)) * **deps:** Update plugin-sdk for k8s to v0.13.18 ([#3407](cloudquery/cloudquery#3407)) ([81874f7](cloudquery/cloudquery@81874f7)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

yevgenypats requested review from erezrokah and hermanschaaf October 30, 2022 17:12

cq-bot added the cli label Oct 30, 2022

yevgenypats requested a review from a team October 30, 2022 17:18

cq-bot added the postgresql label Oct 30, 2022

yevgenypats requested a review from disq as a code owner October 30, 2022 17:56

cq-bot added aws labels Oct 30, 2022

yevgenypats mentioned this pull request Oct 30, 2022

feat: Add CQ type system to support multiple destinations cloudquery/plugin-sdk#320

Merged

cq-bot added k8s labels Oct 31, 2022

hermanschaaf reviewed Oct 31, 2022

View reviewed changes

erezrokah requested changes Oct 31, 2022

View reviewed changes

cq-bot added the heroku label Oct 31, 2022

yevgenypats force-pushed the feat/cq_type_system branch from 8b7579e to 9a86945 Compare October 31, 2022 16:16

erezrokah reviewed Oct 31, 2022

View reviewed changes

plugins/destination/postgresql/CHANGELOG.md Outdated Show resolved Hide resolved

yevgenypats requested review from erezrokah and hermanschaaf and removed request for disq October 31, 2022 16:41

hermanschaaf reviewed Oct 31, 2022

View reviewed changes

plugins/destination/postgresql/client/migrate.go Show resolved Hide resolved

hermanschaaf reviewed Oct 31, 2022

View reviewed changes

plugins/destination/postgresql/client/migrate.go Show resolved Hide resolved

erezrokah reviewed Oct 31, 2022

View reviewed changes

plugins/source/digitalocean/docs/tables/digitalocean_domain_records.md Show resolved Hide resolved

erezrokah reviewed Oct 31, 2022

View reviewed changes

plugins/source/azure/docs/tables/azure_cdn_custom_domains.md Show resolved Hide resolved

fix float8

ac2e273

yevgenypats added the automerge Automatically merge once required checks pass label Nov 1, 2022

fix float bug

bb3ed1a

yevgenypats merged commit f265a94 into main Nov 1, 2022

yevgenypats deleted the feat/cq_type_system branch November 1, 2022 08:04

Conversation

yevgenypats commented Oct 30, 2022

Uh oh!

hermanschaaf Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

yevgenypats Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

hermanschaaf Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

yevgenypats Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

hermanschaaf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erezrokah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants