feat!: Remove drift by disq · Pull Request #887 · cloudquery/cloudquery

disq · 2022-06-13T16:04:52Z

Continued from #752

cq init should generate yaml
after config parse, actions must be taken to set/reinit values from config
make sure json-schema passes
all providers will need to change, as example configs are hcl
sdk needs to be updated (GetProviderConfig)
~~policies will need updating/converting~~
cq init should get yaml config from providers and correctly inject it?

bbernays · 2022-06-13T16:09:06Z

I don't think we should be removing hcl support in the same PR we are adding yaml support. This would allow users to transition over at their own pace, while also lowering the testing bar for us as we won't be breaking any functionality

disq · 2022-06-13T17:29:49Z

I don't think we should be removing hcl support in the same PR we are adding yaml support. This would allow users to transition over at their own pace, while also lowering the testing bar for us as we won't be breaking any functionality

Why not introduce a converter? Otherwise the code gets very complicated (especially provider config logic in the SDK)

yevgenypats · 2022-06-13T17:59:54Z

I also not sure about having two code bases and bug types to support - what is the value of customers upgrading at their own pace? eventually we will have to deprecate anyway. I think converter might be a better take and even that I think it should be a big deal to run cloudquery init aws and convert manually - I think given in terms of where we are in terms of users this should be ok to introduce such breaking change as long as we have good documentation, explanation and it's a major version etc...

disq · 2022-06-16T21:04:21Z

go run main.go init test --config config.yml results:

cloudquery:
    providers:
        - name: test
          version: latest
    connection:
        type: postgres
        username: postgres
        password: pass
        host: localhost
        port: 15432
        database: postgres
        sslmode: disable
providers:
    # provider configurations
    test:
        account:
            name: "1"
            id: testid
        regions:
            - asdas
        resources:
            # list of resources to fetch
            - slow_resource
            - very_slow_resource
            - error_resource
            - panic_resource
            - migrate_resource

moving the comments above the list entries is complicated, not sure if it's even possible without getting dirty (dirtier). The problem is, yaml.Node is good for reading data but when you encode it on the root level (if you don't use a map to hold your root keys, like I was forced to do after many iterations) the results aren't pretty, it simply marshals the whole meta-struct and not just the contents.

yevgenypats · 2022-06-17T06:42:07Z

Nice! Going to run locally and play with so I can give a more thorough review but one nit in the meanwhile. I think given it will be a breaking change the default should be the yaml generator and if someone want to use the old hcl format he will need to use the flag. This way more users will migrate faster and we can deprecate hcl faster.

roneli

Looks good, two comments :)

pkg/config/config_parser.go

`go run main.go init aws --config config.yml`

This reverts commit 676f44b.

🤖 I have created a release *beep* *boop* --- ## [0.27.0-rc1](v0.26.4...v0.27.0-rc1) (2022-06-21) ### ⚠ BREAKING CHANGES * Remove drift (#887) ### Features * Remove drift ([#887](#887)) ([3d387bd](3d387bd)) ### Miscellaneous Chores * Release 0.27.0-rc1 ([#962](#962)) ([3a2ec6d](3a2ec6d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [0.27.0](v0.26.4...v0.27.0) (2022-06-22) ### ⚠ BREAKING CHANGES * Remove drift (#887) ### Features * Remove drift ([#887](#887)) ([3d387bd](3d387bd)) * Add YAML configuration support ([#887](#887)) ([3d387bd](3d387bd)) ### Bug Fixes * Default alias to name ([#966](#966)) ([706447c](706447c)) * Support modules tag in config ([#965](#965)) ([379344f](379344f)) * **deps:** Update module github.com/cloudquery/cq-provider-sdk to v0.12.1 ([#972](#972)) ([1f871e9](1f871e9)) * Request correct config format (YAML) from provider ([#968](#968)) ([999b68d](999b68d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

to go with cloudquery/cq-provider-sdk#332 and cloudquery/cloudquery#887 `go run main.go init azure --config config.yml` creates this: ```yaml cloudquery: providers: - name: azure version: latest connection: type: postgres username: postgres password: pass host: localhost port: 5432 database: postgres sslmode: disable providers: # provider configurations - name: azure # Optional. if you not specified, cloudquery tries to access all subscriptions available to tenant # subscriptions: # - <YOUR_SUBSCRIPTION_ID_HERE> # # list of resources to fetch resources: - account.locations - authorization.role_assignments #... ```

to go with cloudquery/cq-provider-sdk#332 and #887 `go run main.go init okta --config config.yml` creates this: ```yaml cloudquery: providers: - name: okta version: latest connection: type: postgres username: postgres password: pass host: localhost port: 5432 database: postgres sslmode: disable providers: # provider configurations - name: okta # Optional. Okta Token to access API, you can set this with OKTA_API_TOKEN env variable # token = <YOUR_OKTA_TOKEN> # Required. You okta domain name # domain = https://<CHANGE_THIS_TO_YOUR_OKTA_DOMAIN>.okta.com # # list of resources to fetch resources: - users ```

to go with cloudquery/cq-provider-sdk#332 and #887 `go run main.go init azure --config config.yml` creates this: ```yaml cloudquery: providers: - name: azure version: latest connection: type: postgres username: postgres password: pass host: localhost port: 5432 database: postgres sslmode: disable providers: # provider configurations - name: azure # Optional. if you not specified, cloudquery tries to access all subscriptions available to tenant # subscriptions: # - <YOUR_SUBSCRIPTION_ID_HERE> # # list of resources to fetch resources: - account.locations - authorization.role_assignments #... ```

disq requested review from a team and bbernays and removed request for a team June 13, 2022 16:04

disq marked this pull request as draft June 13, 2022 16:05

disq force-pushed the feat/hcl-to-yaml-config branch 5 times, most recently from 12fd8cd to 87356fc Compare June 16, 2022 16:17

disq marked this pull request as ready for review June 16, 2022 21:04

disq mentioned this pull request Jun 16, 2022

feat: YAML config support cloudquery/cq-provider-aws#1067

Merged

disq requested review from roneli and yevgenypats June 16, 2022 21:35

yevgenypats approved these changes Jun 17, 2022

View reviewed changes

erezrokah mentioned this pull request Jun 19, 2022

refactor: Split configuration parsing into steps #913

Merged

5 tasks

roneli approved these changes Jun 19, 2022

View reviewed changes

pkg/config/config_parser.go Show resolved Hide resolved

pkg/config/config_parser.go Outdated Show resolved Hide resolved

disq force-pushed the feat/hcl-to-yaml-config branch 2 times, most recently from e659ad7 to 7aa7905 Compare June 20, 2022 08:37

This was referenced Jun 20, 2022

feat: Change HCL to YAML #711

Closed

feat: YAML config support cloudquery/cq-provider-gcp#352

Merged

disq force-pushed the feat/hcl-to-yaml-config branch from 5daf4a2 to d5096bf Compare June 20, 2022 15:50

disq changed the title ~~feat!: Remove HCL in favor of YAML, remove drift~~ feat!: Remove drift Jun 21, 2022

disq added 3 commits June 21, 2022 09:39

feat!: Remove HCL in favor of YAML, remove drift first

1320ec8

Move things around, add yaml/hcl struct tags

6e39c7c

Handle config

6db88d8

disq added 16 commits June 21, 2022 09:39

try

02709c7

cq init

ae247c0

`go run main.go init aws --config config.yml`

clean up

86b0508

"fix" test

c4f6b7f

make it somewhat work

c3c233a

make fetch work

ab963f1

Use config.yml by default, unless config.hcl already exists

9df0928

clean up

e5103d3

update SDK

c7e417f

update SDK

c61ee78

cq init: unsightly way of generating yaml

5dabbe4

update SDK

0cc251e

Revert "cq init: unsightly way of generating yaml"

fa80e68

This reverts commit 676f44b.

sdk

275da08

default to hcl

e86a4dd

update SDK

b6d9445

disq force-pushed the feat/hcl-to-yaml-config branch from 40f15ef to b6d9445 Compare June 21, 2022 08:46

disq merged commit 3d387bd into cloudquery:main Jun 21, 2022

disq deleted the feat/hcl-to-yaml-config branch June 21, 2022 08:51

cq-bot mentioned this pull request Jun 21, 2022

chore(main): Release v0.27.0-rc1 #961

Merged

disq mentioned this pull request Jun 21, 2022

feat: YAML config support cloudquery/cq-provider-azure#353

Merged

disq mentioned this pull request Jun 21, 2022

feat: YAML config support cloudquery/cq-provider-k8s#182

Merged

This was referenced Jun 21, 2022

chore(main): Release v0.27.0-rc2 #964

Merged

chore(main): Release v0.27.0 #977

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Remove drift#887

feat!: Remove drift#887
disq merged 19 commits intocloudquery:mainfrom
disq:feat/hcl-to-yaml-config

disq commented Jun 13, 2022 •

edited

Loading

Uh oh!

bbernays commented Jun 13, 2022

Uh oh!

disq commented Jun 13, 2022

Uh oh!

yevgenypats commented Jun 13, 2022

Uh oh!

disq commented Jun 16, 2022

Uh oh!

yevgenypats commented Jun 17, 2022

Uh oh!

roneli left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

disq commented Jun 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bbernays commented Jun 13, 2022

Uh oh!

disq commented Jun 13, 2022

Uh oh!

yevgenypats commented Jun 13, 2022

Uh oh!

disq commented Jun 16, 2022

Uh oh!

yevgenypats commented Jun 17, 2022

Uh oh!

roneli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

disq commented Jun 13, 2022 •

edited

Loading