feat(azure): Move to new sdk by yevgenypats · Pull Request #5350 · cloudquery/cloudquery

yevgenypats · 2022-12-04T16:49:05Z

This PR comes instead of this #5286 as old pr got messed up.

This generates the autogeneration code and tries to generate the whole huge Azure SDK completely.

BEGIN_COMMIT_OVERRIDE
feat(azure): Move to new sdk

BREAKING CHANGE: Move to new SDK. Many new resources were added, please see https://www.cloudquery.io/docs/plugins/sources/azure/tables for the new list. The main difference in columns is that we don't unwrap Azure properties into separate columns. Instead properties are stored as a JSON column
END_COMMIT_OVERRIDE

plugins/source/azure/resources/services/armaad/private_link_policy.go

plugins/source/azure/resources/services/armaad/private_link_policy_mock_test.go

plugins/source/azure/resources/services/armapimanagement/group_contract.go

plugins/source/azure/resources/services/armbatch/account.go

vercel · 2022-12-06T08:27:51Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
cloudquery-web	⬜️ Ignored (Inspect)		Dec 6, 2022 at 0:24AM (UTC)

yevgenypats · 2022-12-11T08:03:49Z

Closes #5425
Closes #5293

erezrokah

Thanks for the follow up @yevgenypats the new code gen changes look great 🚀
I think the switch to the new SDK removed a lot of relations we had as we multiplex on resource group instead of getting it from the parent, for example:

cloudquery/plugins/source/azure/resources/services/network/virtual_network_gateway_connections.go

Line 180 in 2ce06fc

    
           response, err := svc.ListConnections(ctx, resourceDetails.ResourceGroup, *gateway.Name)

vs

cloudquery/plugins/source/azure/resources/services/network/virtual_network_gateway_connections.go

Line 65 in d5dace5

pager := svc.NewListPager(cl.ResourceGroup, nil)

WHICH IS GREAT 🎸

Added a few non blocking comments on the code.
Also a few comments similar to the ones I left on @candiduslynx PR:

subscription_id is missing from all the tables. Do we want to re-add it?
Relations are missing a link to the parent, for example azure_cdn_endpoints had profile_id column.
In the current plugin there's a convention to use lowercase table methods, e.g. endpoints instead of Endpoints for relation tables to prevent them being called from outside the package accidentally (we had a few bugs related to this in v0). Do we want to keep that convention? Since everything is auto generated I wouldn't worry to much if it's difficult to add.

Asking again - can we remove codegen1/recipes/ from source code? It's an intermediate directory if I'm not mistaken and is not required for make gen to work or the code to compile correct?

Finally I think we still to do a pass on the relations that are missing, for example

cloudquery/plugins/source/azure/resources/services/sql/servers.go

Line 105 in 2ce06fc

firewallRules(),

Approving since I think it will be easier to address the issues in additional PRs

plugins/source/azure/client/client.go

erezrokah · 2022-12-11T10:28:27Z

plugins/source/azure/client/helpers.go

-	ResourceName  string
-}
+// this is used to hook ParseResourceGroup and to have easier codegen
+var debug = false


Looks like this is used in the tests. Maybe add an option API to ParseResourceGroup to return a mock response? Or even rename this from debug to returnMockResourceGroup

plugins/source/azure/client/multiplexer.go

plugins/source/azure/codegen0/internal/azparser/parser.go

plugins/source/azure/codegen0/internal/azparser/gomod.go

erezrokah · 2022-12-11T10:40:17Z

plugins/source/azure/codegen0/internal/azparser/parser.go

+	for _, t := range tables {
+		// skip tables witout URL (or at least that we didn't find one)
+		// not NewListPager struct and more than 3 params
+		if t.URL == "" || !t.HasListPager || !isArrayExist(supportedNewListPagerParams, t.NewListPagerParams) {


Not sure, but is one of the conditions here consists as a warning? Maybe no URL. You can ignore if this comment is not relevant

I've just pushed/simplified the code here to support multiple types of pager in codegen0

plugins/source/azure/codegen0/internal/azparser/parser.go

plugins/source/azure/codegen0/main.go

plugins/source/azure/codegen2/recipes/resources_e.go

Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

yevgenypats · 2022-12-11T13:38:11Z

Thanks for the follow up @yevgenypats the new code gen changes look great 🚀 I think the switch to the new SDK removed a lot of relations we had as we multiplex on resource group instead of getting it from the parent, for example:

cloudquery/plugins/source/azure/resources/services/network/virtual_network_gateway_connections.go

Line 180 in 2ce06fc

response, err := svc.ListConnections(ctx, resourceDetails.ResourceGroup, *gateway.Name)

vs

cloudquery/plugins/source/azure/resources/services/network/virtual_network_gateway_connections.go

Line 65 in d5dace5

pager := svc.NewListPager(cl.ResourceGroup, nil)

WHICH IS GREAT 🎸

Added a few non blocking comments on the code. Also a few comments similar to the ones I left on @candiduslynx PR:

subscription_id is missing from all the tables. Do we want to re-add it?

Relations are missing a link to the parent, for example azure_cdn_endpoints had profile_id column.

In the current plugin there's a convention to use lowercase table methods, e.g. endpoints instead of Endpoints for relation tables to prevent them being called from outside the package accidentally (we had a few bugs related to this in v0). Do we want to keep that convention? Since everything is auto generated I wouldn't worry to much if it's difficult to add.

Asking again - can we remove codegen1/recipes/ from source code? It's an intermediate directory if I'm not mistaken and is not required for make gen to work or the code to compile correct?

Finally I think we still to do a pass on the relations that are missing, for example

cloudquery/plugins/source/azure/resources/services/sql/servers.go

Line 105 in 2ce06fc

firewallRules(),

Approving since I think it will be easier to address the issues in additional PRs

Thanks. 1 and 3 addressed. 2 will be addressed in follow-up PRs.

erezrokah · 2022-12-11T13:53:55Z

We should have merged this as a breaking change, but I think we can tweak the changelog later on

cq-bot added the azure label Dec 4, 2022

candiduslynx suggested changes Dec 5, 2022

View reviewed changes

This was referenced Dec 5, 2022

Azure Load Balancers #5293

Closed

Azure DNS resources (zones and records) #5126

Closed

yevgenypats marked this pull request as ready for review December 6, 2022 10:54

yevgenypats requested a review from erezrokah as a code owner December 6, 2022 10:54

yevgenypats added 21 commits December 7, 2022 14:30

feat: Azure autogenerate autogeneration code

81ffd7c

add mockgen

34a51c8

more progress

9bfd79c

some more progress with tests passing

948bef7

more wip

399f4de

some more progress

0ff0c03

more work

e153db5

more wip tests passing

ee7e805

more wip

9bbd78f

tidy

c2733b7

more wip

4f22aff

tests passing

07a8f71

more formatting

75e10c5

lint passing

219cfd5

add pk

2f466c8

some more improvments of not calling not enabled apis

332b59d

working azure plugin

17b23f3

tests passing

109ce0f

docs

a4cdf37

sort lexographically

c2947f6

added some more docs

a3e16b7

yevgenypats force-pushed the feat/meta_gen_azure branch from 45b6825 to a3e16b7 Compare December 7, 2022 12:32

yevgenypats requested a review from hermanschaaf December 7, 2022 12:33

yevgenypats added 4 commits December 10, 2022 23:19

minor fixes

b6206dc

fmt fix

795c142

simplify codegen more

d5dace5

add tables

0fee0da

erezrokah mentioned this pull request Dec 11, 2022

Add Azure Resource_skus and add its table also #5521

Closed

erezrokah approved these changes Dec 11, 2022

View reviewed changes

yevgenypats and others added 11 commits December 11, 2022 14:46

more work

1b53586

tidy more

baaada9

Update plugins/source/azure/client/client.go

7dc70e9

Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

Update plugins/source/azure/codegen0/internal/azparser/parser.go

03b64b1

Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

Update plugins/source/azure/codegen0/internal/azparser/parser.go

a8cfa42

Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

Update plugins/source/azure/codegen0/internal/azparser/gomod.go

ce2ff3b

Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

Update plugins/source/azure/codegen0/main.go

2685eb1

Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>

fmt

8629215

minor fixes

6b80847

make relations private

a41200e

add subscription_id

ac35f8d

gen docs

1fb5de0

vercel bot deployed to Preview December 11, 2022 13:42 View deployment

Merge branch 'main' into feat/meta_gen_azure

9f50e35

vercel bot deployed to Preview December 11, 2022 13:47 View deployment

yevgenypats merged commit 8657395 into main Dec 11, 2022

yevgenypats deleted the feat/meta_gen_azure branch December 11, 2022 13:48

cq-bot mentioned this pull request Dec 11, 2022

chore(main): Release plugins-source-azure v2.0.0 #5220

Merged

This was referenced Dec 13, 2022

Azure Table Storage #5425

Closed

web app configurations table #5542

Closed

candiduslynx mentioned this pull request Dec 30, 2022

Azure source plugin is missing azure_storage_blob_services table #6157

Closed

Conversation

yevgenypats commented Dec 4, 2022 • edited by erezrokah Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vercel bot commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yevgenypats commented Dec 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erezrokah left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erezrokah Dec 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erezrokah Dec 11, 2022

Choose a reason for hiding this comment

Uh oh!

yevgenypats Dec 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yevgenypats commented Dec 11, 2022

Uh oh!

erezrokah commented Dec 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yevgenypats commented Dec 4, 2022 •

edited by erezrokah

Loading

vercel bot commented Dec 6, 2022 •

edited

Loading

yevgenypats commented Dec 11, 2022 •

edited

Loading

erezrokah left a comment •

edited

Loading