feat(postgresql): Move to standalone plugin#2043
Closed
yevgenypats wants to merge 25 commits intomainfrom
Closed
Conversation
5a1a5f0 to
a9ed9b1
Compare
yevgenypats
added a commit
to cloudquery/plugin-sdk
that referenced
this pull request
Sep 25, 2022
erezrokah
requested changes
Sep 25, 2022
Member
erezrokah
left a comment
There was a problem hiding this comment.
I think once we fix the CI errors this should be a good start and we can do the release changes in a follow up PR.
Also we should delete the binary plugins/destination/postgresql/postgresql and ignore it.
| matrix: ${{ fromJson(needs.resolve-modules.outputs.matrix) }} | ||
| fail-fast: false | ||
| runs-on: ubuntu-latest | ||
| services: |
Member
There was a problem hiding this comment.
Don't we need this to test that we successfully write to the database?
Contributor
Author
There was a problem hiding this comment.
I think let's discuss this offline as it's a bigger discussion but every destination plugin will have different requirements for databases and so on (sources dont need any db). so I think we can't automate much here but rather just have one workflow per destination plugin (and I dont think we will have more than 10 so shouldn't be an issue or a lot to automate) even if we will have to copy paste a bit
Member
There was a problem hiding this comment.
Maybe keep the tests at least until we have the discussion? Otherwise we won't have coverage for the sync command
Comment on lines
+12
to
+13
| // https://github.com/cloudquery/cloudquery/releases/download/plugins-source-test-v1.1.0/test_darwin_arm64.zip | ||
| // https://github.com/cloudquery/cloudquery/releases/download/plugins-source-test-v1.1.5/test_darwin_arm64.zip No newline at end of file |
Member
There was a problem hiding this comment.
Can we delete these comments?
yevgenypats
added a commit
to cloudquery/plugin-sdk
that referenced
this pull request
Sep 25, 2022
This should go with cloudquery/cloudquery#2043
6 tasks
kodiakhq bot
pushed a commit
that referenced
this pull request
Sep 25, 2022
#### Summary Preparing to #2043 ---
hermanschaaf
approved these changes
Sep 26, 2022
Contributor
hermanschaaf
left a comment
There was a problem hiding this comment.
LGTM, approving with some minor nit comments
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
#### Summary Simple enough to do cc @bernays assuming we want this. This creates the following structure (for example) in each GCP zip: ``` ├── plugins │ └── source │ └── gcp └── policies ├── README.md ├── cis_v1.2.0 │ ├── policy.sql │ ├── section_1.sql │ ├── section_2.sql │ ├── section_3.sql │ ├── section_4.sql │ ├── section_5.sql │ ├── section_6.sql │ └── section_7.sql ├── create_gcp_policy_results.sql ├── policy.sql ├── queries │ ├── bigquery │ │ ├── datasets_publicly_accessible.sql │ │ ├── datasets_without_default_cmek.sql │ │ └── tables_not_encrypted_with_cmek.sql │ ├── compute │ │ ├── allow_traffic_behind_iap.sql │ │ ├── default_network_exist.sql │ │ ├── disks_encrypted_with_csek.sql │ │ ├── dnssec_disabled.sql │ │ ├── flow_logs_disabled_in_vpc.sql │ │ ├── instance_ip_forwarding_enabled.sql │ │ ├── instances_with_default_service_account.sql │ │ ├── instances_with_default_service_account_with_full_access.sql │ │ ├── instances_with_public_ip.sql │ │ ├── instances_with_shielded_vm_disabled.sql │ │ ├── instances_without_block_project_wide_ssh_keys.sql │ │ ├── instances_without_confidential_computing.sql │ │ ├── legacy_network_exist.sql │ │ ├── oslogin_disabled.sql │ │ ├── rdp_access_permitted.sql │ │ ├── serial_port_connection_enabled.sql │ │ ├── ssh_access_permitted.sql │ │ └── ssl_proxy_with_weak_cipher.sql │ ├── dns │ │ ├── key_signing_with_rsasha1.sql │ │ └── zone_signing_with_rsasha1.sql │ ├── iam │ │ ├── kms_separation_of_duties.sql │ │ ├── managed_service_account_keys.sql │ │ ├── separation_of_duties.sql │ │ ├── service_account_admin_priv.sql │ │ ├── service_account_keys_not_rotated.sql │ │ └── users_with_service_account_token_creator_role.sql │ ├── kms │ │ ├── keys_not_rotated_within_90_days.sql │ │ └── publicly_accessible.sql │ ├── logging │ │ ├── audit_config_changes_without_log_metric_filter_alerts.sql │ │ ├── custom_role_changes_without_log_metric_filter_alerts.sql │ │ ├── dns_logging_disabled.sql │ │ ├── log_buckets_retention_policy_disabled.sql │ │ ├── not_configured_across_services_and_users.sql │ │ ├── project_ownership_changes_without_log_metric_filter_alerts.sql │ │ ├── sinks_not_configured_for_all_log_entries.sql │ │ ├── sql_instance_changes_without_log_metric_filter_alerts.sql │ │ ├── storage_iam_changes_without_log_metric_filter_alerts.sql │ │ ├── vpc_firewall_changes_without_log_metric_filter_alerts.sql │ │ ├── vpc_network_changes_without_log_metric_filter_alerts.sql │ │ └── vpc_route_changes_without_log_metric_filter_alerts.sql │ ├── manual.sql │ ├── sql │ │ ├── db_instance_publicly_accessible.sql │ │ ├── db_instance_with_public_ip.sql │ │ ├── db_instance_without_ssl.sql │ │ ├── db_instances_without_backups.sql │ │ ├── mysql_local_inline_flag_on.sql │ │ ├── mysql_skip_show_database_flag_off.sql │ │ ├── postgresql_log_checkpoints_flag_off.sql │ │ ├── postgresql_log_connections_flag_off.sql │ │ ├── postgresql_log_disconnections_flag_off.sql │ │ ├── postgresql_log_duration_flag_off.sql │ │ ├── postgresql_log_error_verbosity_flag_not_strict.sql │ │ ├── postgresql_log_executor_stats_flag_on.sql │ │ ├── postgresql_log_hostname_flag_off.sql │ │ ├── postgresql_log_lock_waits_flag_off.sql │ │ ├── postgresql_log_min_duration_statement_flag_on.sql │ │ ├── postgresql_log_min_error_statement_flag_less_error.sql │ │ ├── postgresql_log_parser_stats_flag_on.sql │ │ ├── postgresql_log_planner_stats_flag_on.sql │ │ ├── postgresql_log_statement_stats_flag_on.sql │ │ ├── postgresql_log_temp_files_flag_off.sql │ │ ├── sqlserver_contained_database_authentication_flag_on.sql │ │ ├── sqlserver_cross_db_ownership_chaining_flag_on.sql │ │ ├── sqlserver_external_scripts_enabled_flag_on.sql │ │ ├── sqlserver_remote_access_flag_on.sql │ │ ├── sqlserver_trace_flag_on.sql │ │ ├── sqlserver_user_connections_flag_not_set.sql │ │ └── sqlserver_user_options_flag_set.sql │ └── storage │ ├── buckets_publicly_accessible.sql │ └── buckets_without_uniform_bucket_level_access.sql └── views ├── buckets_permissions.sql ├── firewall_allowed_rules.sql ├── log_metric_filters.sql └── project_policy_members.sql ``` ---
…2044) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/cloudquery/plugin-sdk](https://togithub.com/cloudquery/plugin-sdk) | require | minor | `v0.7.13` -> `v0.8.2` | | [github.com/cloudquery/plugin-sdk](https://togithub.com/cloudquery/plugin-sdk) | require | patch | `v0.8.1` -> `v0.8.2` | --- <details> <summary>cloudquery/plugin-sdk</summary> [`v0.8.2`](https://togithub.com/cloudquery/plugin-sdk/releases/tag/v0.8.2) [Compare Source](https://togithub.com/cloudquery/plugin-sdk/compare/v0.8.1...v0.8.2) - Fix typo in ValueTypeFromString ([#​172](https://togithub.com/cloudquery/plugin-sdk/issues/172)) ([12cb9c9](https://togithub.com/cloudquery/plugin-sdk/commit/12cb9c9b9ee24dd0282da35926229d1256f11696)) [`v0.8.1`](https://togithub.com/cloudquery/plugin-sdk/releases/tag/v0.8.1) [Compare Source](https://togithub.com/cloudquery/plugin-sdk/compare/v0.8.0...v0.8.1) - **codegen:** Add `WithResolverTransformer` option ([#​164](https://togithub.com/cloudquery/plugin-sdk/issues/164)) ([9529956](https://togithub.com/cloudquery/plugin-sdk/commit/95299560af85a687d1e7274ab80541e02948980a)) [`v0.8.0`](https://togithub.com/cloudquery/plugin-sdk/releases/tag/v0.8.0) [Compare Source](https://togithub.com/cloudquery/plugin-sdk/compare/v0.7.13...v0.8.0) - Remove ExampleConfig from client,servers and protobuf ([#​167](https://togithub.com/cloudquery/plugin-sdk/issues/167)) - Remove ExampleConfig from client,servers and protobuf ([#​167](https://togithub.com/cloudquery/plugin-sdk/issues/167)) ([23b1575](https://togithub.com/cloudquery/plugin-sdk/commit/23b15758158318b0bfbad78a344a5d4e2369cf98)) </details> --- 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4xODUuMyIsInVwZGF0ZWRJblZlciI6IjMyLjE4NS4zIn0=--> Co-authored-by: Renovate Bot <renovate@whitesourcesoftware.com>
Fix redirects to plugins, also uses `source` as to be consistent with other routes where the page title matches the route
Preparing to #2043 ---
This should make all versions versioned as `v1`
🤖 I have created a release *beep* *boop* --- ## [0.1.0](plugins-destination-postgresql-v0.0.1...plugins-destination-postgresql-v0.1.0) (2022-09-25) ### Features * Update log ([#2051](#2051)) ([6efdf1c](6efdf1c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Updates the `destination-postgresql` plugin latest version to v0.1.0
🤖 I have created a release *beep* *boop* --- ## [1.0.0-pre.0](cli-v0.33.4-pre.0...cli-v1.0.0-pre.0) (2022-09-25) ### ⚠ BREAKING CHANGES * **cli:** Remove gen command (#2022) ### Features * **cli:** Remove gen command ([#2022](#2022)) ([83a32dd](83a32dd)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk to v0.8.0 ([#1997](#1997)) ([4fa40da](4fa40da)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v0.8.1 ([#2024](#2024)) ([8f88de4](8f88de4)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v0.8.2 ([#2044](#2044)) ([9b69b46](9b69b46)) * Don't print skip download message to console ([#2008](#2008)) ([a947d44](a947d44)) * Use uppercase downloading during progress ([#2006](#2006)) ([e6a7a44](e6a7a44)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
…n't existed (#2056) #### Summary The url for plugins doesn't existed(https://www.cloudquery.io/plugins) and shows 404. ### Changes I update the links from `https://www.cloudquery.io/plugins` to `https://www.cloudquery.io/docs/plugins/source`
Adds migration guides for v1 plugins. Guides added in this PR: - [AWS](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/aws/docs/v1-migration.md) (Updated) - [Azure](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/azure/docs/v1-migration.md) - [Cloudflare](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/cloudflare/docs/v1-migration.md) - [DigitalOcean](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/digitalocean/docs/v1-migration.md) - [GCP](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/gcp/docs/v1-migration.md) - [Github](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/github/docs/v1-migration.md) - [Okta](https://github.com/cloudquery/cloudquery/blob/migration-guides/plugins/source/okta/docs/v1-migration.md) Heroku was left out because it was never released pre-v1. K8s will be added when it is ready, and terraform requires some changes to work (will follow up later today). Generating these requires a bit of manual setup, but for future reference, the process I have locally right now can be replicated using something like this: ``` git worktree add ../v0 git worktree add ../v1 cd ../v0 && git checkout ab65d1e # or any v0 commit cd ../v1 && make build cd .. ``` Then run this script from outside the git root: ``` plugins="aws gcp azure cloudflare digitalocean github okta" mkdir -p docs/tables-v0 mkdir -p docs/tables-v1 for p in $plugins do echo "Generating migration guide for $p" cp -r v0/plugins/source/$p/docs/tables/* docs/tables-v0 ./v1/bin/$p doc docs/tables-v1 ./v1/bin/v1-migration -o cloudquery/plugins/source/$p/docs/v1-migration.md -v1 docs/tables-v0 -v2 docs/tables-v1 done ```
#### Summary <!-- Explain what problem this PR addresses --> ---
Adding a new test destination for CLI testing purposes. The new
destination plugin is basically a dev null.
Also, Im proposing/trying out a .yml per plugin ? maybe this way we can
take advantage of the built-in github
```
paths:
- "plugins/destination/test/**"
```
For destination plugins I think this is specifaclly important because
the workflows vary very much between each plugin as we need different
databases and so on.
I think for source plugin this will also start becoming a thing when we
introduce some smoke tests that will require credentials and so on.
🤖 I have created a release *beep* *boop* --- ## [1.1.0](plugins-destination-test-v1.0.0...plugins-destination-test-v1.1.0) (2022-09-26) ### Features * **dest-test:** Add new test destination ([#2057](#2057)) ([d2e1df2](d2e1df2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
yevgenypats
added a commit
that referenced
this pull request
Sep 26, 2022
This PR got messed up in conflicts #2043 Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following changes in this PR:
Moving PostgreSQL to standalone plugin so we can test the workflow of our destinations. We don't want to support at the moment compiled destinations as we want to use and test one workflow only.
Also, I've simplified the plugin manager and downloader given we use convention over configuration and this code should mostly not changed.
Removed unit-tests from versions and made them pure functions for the following reasons:
go test ./...then we should just add tags to the integration tests (if someone is working on the plugin manager they will need to run integration tests anyway so again not alot of value in maintaing unit-tests for this component imo).