feat: Move azure to avoid codegen. by yevgenypats · Pull Request #6276 · cloudquery/cloudquery

yevgenypats · 2023-01-03T12:37:53Z

This shouldn't introduce any public facing changes.

This should go together with: cloudquery/plugin-sdk#564

In general codegen creates a sub-optimal developer experience where you have to juggle between codegen and code to make any code-changes. Not only it creates a tough experience for the developer it also makes it hard to review and to contribute.

This take here moves codegen only to the discovery phase and moves the transformation to be part of the SDK which will eventually deprecate the codegen package from the SDK completely and will avoid having two APIs which are almost identical but quite the same.

github-actions · 2023-01-03T16:08:53Z

This PR has the following changes to source plugin(s) tables:

Table azure_advisor_recommendations: column order changed for properties
Table azure_advisor_suppressions: column order changed for properties
Table azure_analysisservices_servers: column order changed for id
Table azure_apimanagement_service: column order changed for id
Table azure_appcomplianceautomation_reports: column order changed for properties
Table azure_appconfiguration_configuration_stores: column order changed for id
Table azure_applicationinsights_components: column order changed for id
Table azure_applicationinsights_web_tests: column order changed for id
Table azure_appservice_certificate_orders: column order changed for id
Table azure_appservice_certificates: column order changed for id
Table azure_appservice_deleted_web_apps: column order changed for id
Table azure_appservice_domains: column order changed for id
Table azure_appservice_environments: column order changed for id
Table azure_appservice_plans: column order changed for id
Table azure_appservice_recommendations: column order changed for id
Table azure_appservice_resource_health_metadata: column order changed for id
Table azure_appservice_static_sites: column order changed for id
Table azure_appservice_top_level_domains: column order changed for id
Table azure_appservice_web_apps: column order changed for id
Table azure_authorization_provider_operations_metadata: column order changed for display_name
Table azure_authorization_role_assignments: column order changed for properties
Table azure_authorization_role_definitions: column added with name subscription_id and type String
Table azure_authorization_role_definitions: column order changed for properties
Table azure_automation_account: column order changed for id
Table azure_azurearcdata_postgres_instances: column order changed for id
Table azure_azurearcdata_sql_managed_instances: column order changed for id
Table azure_azurearcdata_sql_server_instances: column order changed for id
Table azure_azuredata_sql_server_registrations: column order changed for id
Table azure_batch_account: column order changed for id
Table azure_billing_accounts: column order changed for properties
Table azure_billing_enrollment_accounts: column order changed for properties
Table azure_billing_periods: column order changed for properties
Table azure_botservice_bots: column order changed for id
Table azure_cdn_edge_nodes: column order changed for properties
Table azure_cdn_endpoints: column order changed for id
Table azure_cdn_managed_rule_sets: column order changed for id
Table azure_cdn_profiles: column order changed for id
Table azure_cdn_rule_sets: column order changed for properties
Table azure_cdn_security_policies: column order changed for properties
Table azure_cognitiveservices_accounts: column order changed for id
Table azure_cognitiveservices_deleted_accounts: column order changed for id
Table azure_compute_cloud_services: column order changed for id
Table azure_compute_disk_accesses: column order changed for id
Table azure_compute_disk_encryption_sets: column order changed for id
Table azure_compute_disks: column order changed for id
Table azure_compute_galleries: column order changed for id
Table azure_compute_images: column order changed for id
Table azure_compute_restore_point_collections: column order changed for id
Table azure_compute_skus: column added with name subscription_id and type String
Table azure_compute_snapshots: column order changed for id
Table azure_compute_virtual_machine_scale_sets: column order changed for id
Table azure_compute_virtual_machines: column order changed for id
Table azure_confluent_marketplace_agreements: column order changed for properties
Table azure_connectedvmware_clusters: column order changed for id
Table azure_connectedvmware_datastores: column order changed for id
Table azure_connectedvmware_hosts: column order changed for id
Table azure_connectedvmware_resource_pools: column order changed for id
Table azure_connectedvmware_v_centers: column order changed for id
Table azure_connectedvmware_virtual_machine_templates: column order changed for id
Table azure_connectedvmware_virtual_machines: column order changed for id
Table azure_connectedvmware_virtual_networks: column order changed for id
Table azure_containerinstance_container_groups: column order changed for id
Table azure_containerregistry_registries: column order changed for id
Table azure_containerservice_managed_clusters: column order changed for id
Table azure_containerservice_snapshots: column order changed for id
Table azure_cosmos_database_accounts: column order changed for id
Table azure_cosmos_locations: column order changed for properties
Table azure_cosmos_mongo_db_databases: column order changed for id
Table azure_cosmos_restorable_database_accounts: column order changed for id
Table azure_cosmos_sql_databases: column order changed for id
Table azure_costmanagement_view_queries: column order changed for etag
Table azure_costmanagement_view_queries: column order changed for properties
Table azure_costmanagement_views: column order changed for id
Table azure_customerinsights_hubs: column order changed for id
Table azure_dashboard_grafana: column order changed for id
Table azure_databox_jobs: column order changed for id
Table azure_datadog_marketplace_agreements: column order changed for properties
Table azure_datadog_monitors: column order changed for id
Table azure_datafactory_factories: column order changed for etag
Table azure_datafactory_factories: column order changed for id
Table azure_datamigration_services: column order changed for id
Table azure_desktopvirtualization_host_pools: column order changed for id
Table azure_devhub_workflow: column order changed for id
Table azure_dns_zones: column order changed for id
Table azure_dnsresolver_dns_forwarding_rulesets: column order changed for id
Table azure_dnsresolver_dns_resolvers: column order changed for id
Table azure_elastic_monitors: column order changed for id
Table azure_engagementfabric_accounts: column order changed for id
Table azure_eventgrid_topic_types: column order changed for properties
Table azure_eventhub_namespaces: column order changed for id
Table azure_frontdoor_front_doors: column order changed for id
Table azure_frontdoor_managed_rule_sets: column order changed for id
Table azure_frontdoor_network_experiment_profiles: column order changed for id
Table azure_hanaonazure_sap_monitors: column order changed for id
Table azure_hdinsight_clusters: column order changed for id
Table azure_healthbot_bots: column order changed for id
Table azure_healthcareapis_services: column order changed for id
Table azure_hybridcompute_private_link_scopes: column order changed for id
Table azure_hybriddatamanager_data_managers: column order changed for id
Table azure_keyvault_keyvault_keys: column order changed for properties
Table azure_keyvault_keyvault_managed_hsms: column order changed for id
Table azure_keyvault_keyvault_secrets: column order changed for properties
Table azure_kusto_clusters: column order changed for id
Table azure_logic_workflows: column order changed for id
Table azure_maintenance_configurations: column order changed for id
Table azure_maintenance_public_maintenance_configurations: column order changed for id
Table azure_mariadb_server_configurations: column order changed for properties
Table azure_mariadb_servers: column order changed for id
Table azure_marketplace_private_store: column order changed for properties
Table azure_monitor_log_profiles: column order changed for id
Table azure_monitor_private_link_scopes: column order changed for id
Table azure_monitor_tenant_activity_log_alerts: column order changed for id
Table azure_monitor_tenant_activity_logs: column order changed for id
Table azure_monitor_tenant_activity_logs: column order changed for subscription_id
Table azure_mysql_server_configurations: column order changed for properties
Table azure_mysql_servers: column order changed for id
Table azure_mysqlflexibleservers_servers: column order changed for id
Table azure_network_custom_ip_prefixes: column order changed for extended_location
Table azure_network_ddos_protection_plans: column order changed for id
Table azure_network_interfaces: column order changed for extended_location
Table azure_network_load_balancers: column order changed for extended_location
Table azure_network_public_ip_addresses: column order changed for extended_location
Table azure_network_public_ip_prefixes: column order changed for extended_location
Table azure_network_subscription_network_manager_connections: column order changed for id
Table azure_network_virtual_networks: column order changed for extended_location
Table azure_networkfunction_azure_traffic_collectors_by_subscription: column order changed for id
Table azure_nginx_deployments: column order changed for id
Table azure_notificationhubs_namespaces: column order changed for id
Table azure_operationalinsights_clusters: column order changed for id
Table azure_operationalinsights_workspaces: column order changed for id
Table azure_peering_service_locations: column order changed for properties
Table azure_peering_service_providers: column order changed for properties
Table azure_policy_assignments: column order changed for id
Table azure_policy_data_policy_manifests: column order changed for properties
Table azure_policy_definitions: column order changed for properties
Table azure_policy_exemptions: column order changed for properties
Table azure_policy_set_definitions: column order changed for properties
Table azure_portal_list_tenant_configuration_violations: column order changed for error_message
Table azure_portal_tenant_configurations: column order changed for properties
Table azure_postgresql_servers: column order changed for id
Table azure_postgresqlflexibleservers_servers: column order changed for id
Table azure_postgresqlhsc_server_groups: column order changed for id
Table azure_powerbidedicated_capacities: column order changed for id
Table azure_privatedns_private_zones: column order changed for id
Table azure_providerhub_provider_registrations: column order changed for properties
Table azure_redhatopenshift_open_shift_clusters: column order changed for id
Table azure_redis_caches: column order changed for id
Table azure_relay_namespaces: column order changed for id
Table azure_reservations_reservation: column order changed for id
Table azure_reservations_reservation_order: column order changed for id
Table azure_resources_resources: column order changed for id
Table azure_saas_resources: column order changed for id
Table azure_search_services: column order changed for id
Table azure_security_alerts: column order changed for properties
Table azure_security_alerts_suppression_rules: column order changed for properties
Table azure_security_applications: column order changed for properties
Table azure_security_assessments_metadata: column order changed for properties
Table azure_security_auto_provisioning_settings: column order changed for properties
Table azure_security_automations: column order changed for id
Table azure_security_connectors: column order changed for id
Table azure_security_discovered_security_solutions: column order changed for properties
Table azure_security_external_security_solutions: column order changed for kind
Table azure_security_governance_rule: column order changed for properties
Table azure_security_jit_network_access_policies: column order changed for id
Table azure_security_pricings: column order changed for properties
Table azure_security_regulatory_compliance_standards: column order changed for properties
Table azure_security_secure_score_control_definitions: column order changed for properties
Table azure_security_secure_score_controls: column order changed for properties
Table azure_security_solutions: column order changed for properties
Table azure_security_tasks: column order changed for properties
Table azure_security_workspace_settings: column order changed for properties
Table azure_servicebus_namespaces: column order changed for id
Table azure_sql_instance_pools: column order changed for id
Table azure_sql_managed_instances: column order changed for id
Table azure_sql_servers: column order changed for id
Table azure_sql_virtual_clusters: column order changed for id
Table azure_sqlvirtualmachine_groups: column order changed for id
Table azure_sqlvirtualmachine_sql_virtual_machines: column order changed for id
Table azure_storage_accounts: column order changed for id
Table azure_storage_blob_services: column order changed for properties
Table azure_storage_containers: column order changed for id
Table azure_storage_tables: column order changed for properties
Table azure_storagecache_caches: column order changed for id
Table azure_streamanalytics_streaming_jobs: column order changed for id
Table azure_subscription_subscriptions: column order changed for id
Table azure_support_services: column order changed for properties
Table azure_support_tickets: column order changed for properties
Table azure_synapse_private_link_hubs: column order changed for id
Table azure_synapse_workspaces: column order changed for id
Table azure_windowsesu_multiple_activation_keys: column order changed for id
Table azure_windowsiot_services: column order changed for id
Table azure_workloads_monitors: column order changed for id

plugins/source/azure/resources/services/costmanagement/views.go

erezrokah

Looks good, added a couple of non blocking comments.

I didn't go through everything, so I'm hoping non of the table resolver implementations changed. Might be good to keep those in separate files

plugins/source/azure/resources/services/authorization/role_definitions.go

erezrokah · 2023-01-04T10:10:12Z

plugins/source/azure/resources/services/compute/skus.go

 		Columns: []schema.Column{
+			{
+				Name:     "subscription_id",


Same here, is adding this subscription_id column expected?

This shouldn't introduce any public facing changes.

vercel · 2023-01-04T13:31:55Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
cloudquery-web	🔄 Building (Inspect)		Jan 4, 2023 at 1:31PM (UTC)

Similar to #6276 moving transformation to be part of our SDK cloudquery/plugin-sdk#564 This is related to cloudquery/plugin-sdk#573 Few notes on column changes: most of changes are just additional columns and this should be fine as we did too much transformation. we want to keep the code simple, focus on extraction and provide and easy way of doing "dumb" one level transformation via reflection. The additional column are because we have in some places a duplicate where we add our own `arn` so it can be consistent but this is our decision and we still want to be as close as possible to the API so some duplicates is fine (and it makes the code simple and consistent). breaking changes in `aws_docdb_pending_maintenance_actions ` those apparently never worked and those columns where always empty (they didn't have mock tests so we didn't catch those) Co-authored-by: Kemal <223029+disq@users.noreply.github.com> Co-authored-by: Kemal Hadimli <disq@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [3.2.0](plugins-source-azure-v3.1.0...plugins-source-azure-v3.2.0) (2023-01-10) ### Features * Move azure to avoid codegen. ([#6276](#6276)) ([a0a8f0a](a0a8f0a)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.17.2 ([#6260](#6260)) ([805972a](805972a)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.19.0 ([#6363](#6363)) ([ae6967c](ae6967c)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.20.0 ([#6376](#6376)) ([d6187ec](d6187ec)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.21.0 ([#6382](#6382)) ([5baea40](5baea40)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.22.0 ([#6516](#6516)) ([b7e4e73](b7e4e73)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.23.0 ([#6522](#6522)) ([ce24f1d](ce24f1d)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.24.1 ([#6553](#6553)) ([392b848](392b848)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

yevgenypats requested a review from hermanschaaf January 3, 2023 12:37

yevgenypats requested a review from erezrokah as a code owner January 3, 2023 12:37

cq-bot added the azure label Jan 3, 2023

cq-bot added the website label Jan 3, 2023

vercel bot deployed to Preview January 3, 2023 16:10 View deployment

vercel bot deployed to Preview January 3, 2023 16:30 View deployment

vercel bot deployed to Preview January 3, 2023 16:44 View deployment

disq reviewed Jan 3, 2023

View reviewed changes

plugins/source/azure/resources/services/costmanagement/views.go Show resolved Hide resolved

erezrokah approved these changes Jan 4, 2023

View reviewed changes

yevgenypats mentioned this pull request Jan 4, 2023

feat: Move AWS to use built-in transformations (no codegen) #6337

Merged

yevgenypats added the automerge Automatically merge once required checks pass label Jan 4, 2023

yevgenypats added 8 commits January 4, 2023 15:22

feat: Move azure to avoid codegen.

9592d62

This shouldn't introduce any public facing changes.

add missing tables

6e98506

add missing docs

064a7d8

add more missing tables

bb8ceb5

add all missing again

a4dcb2a

fix costmgmt

9b79dc2

remove etag

c15ea1f

lint

67f557f

yevgenypats force-pushed the feat/azure_no_codegen branch from b9d6fd2 to 67f557f Compare January 4, 2023 13:23

update makefile

b55096f

yevgenypats merged commit a0a8f0a into main Jan 4, 2023

yevgenypats deleted the feat/azure_no_codegen branch January 4, 2023 13:36

cq-bot mentioned this pull request Jan 4, 2023

chore(main): Release plugins-source-azure v3.2.0 #6308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Move azure to avoid codegen.#6276

feat: Move azure to avoid codegen.#6276
yevgenypats merged 9 commits intomainfrom
feat/azure_no_codegen

yevgenypats commented Jan 3, 2023

Uh oh!

github-actions bot commented Jan 3, 2023 •

edited

Loading

Uh oh!

Uh oh!

erezrokah left a comment

Uh oh!

Uh oh!

erezrokah Jan 4, 2023

Uh oh!

vercel bot commented Jan 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yevgenypats commented Jan 3, 2023

Uh oh!

github-actions bot commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR has the following changes to source plugin(s) tables:

Uh oh!

Uh oh!

erezrokah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erezrokah Jan 4, 2023

Choose a reason for hiding this comment

Uh oh!

vercel bot commented Jan 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Jan 3, 2023 •

edited

Loading