Skip to content

feat(destinations): Add migrate_mode#604

Merged
erezrokah merged 2 commits intocloudquery:mainfrom
erezrokah:feat/add_migrate_mode
Jan 15, 2023
Merged

feat(destinations): Add migrate_mode#604
erezrokah merged 2 commits intocloudquery:mainfrom
erezrokah:feat/add_migrate_mode

Conversation

@erezrokah
Copy link
Copy Markdown
Member

@erezrokah erezrokah commented Jan 12, 2023

Summary

Related to cloudquery/cloudquery#6600, needed for cloudquery/cloudquery#6763


Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines 🧑‍🎓
  • Run go fmt to format your code 🖊
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests 🧪
  • Ensure the status checks below are successful ✅

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 12, 2023

⏱️ Benchmark results

Comparing with 37254f3

  • DefaultConcurrencyDFS-2 resources/s: 11,046 ⬆️ 8.32% increase vs. 37254f3
  • DefaultConcurrencyRoundRobin-2 resources/s: 11,161 ⬇️ 11.44% decrease vs. 37254f3
  • Glob-2 ns/op: 157.5 (no change)
  • TablesWithChildrenDFS-2 resources/s: 28,156 ⬇️ 9.88% decrease vs. 37254f3
  • TablesWithChildrenRoundRobin-2 resources/s: 28,298 ⬇️ 0.52% decrease vs. 37254f3
  • TablesWithRateLimitingDFS-2 resources/s: 28.45 ⬆️ 0.91% increase vs. 37254f3
  • TablesWithRateLimitingRoundRobin-2 resources/s: 803.5 ⬇️ 4.49% decrease vs. 37254f3
  • BufferedScanner-2 ns/op: 10.04 ⬆️ 6.45% increase vs. 37254f3
  • LogReader-2 ns/op: 30.82 ⬆️ 0.26% increase vs. 37254f3

Path string `json:"path,omitempty"`
Registry Registry `json:"registry,omitempty"`
WriteMode WriteMode `json:"write_mode,omitempty"`
MigrateMode MigrateMode `json:"migrate_mode,omitempty"`
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can also be a boolean, just found it easier to communicate the purpose using strings

Copy link
Copy Markdown
Contributor

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM; are we also planning on supporting a --migrate forced (or similar) flag for the CLI? Might make it a bit easier for users doing things interactively

@hermanschaaf
Copy link
Copy Markdown
Contributor

We're also not using the setting anywhere yet - is that because it needs to be implemented in each destination? I think we'll need to pass the mode in to the Migrate call so that destinations can act on it?

@erezrokah
Copy link
Copy Markdown
Member Author

a --migrate forced (or similar) flag for the CLI? Might make it a bit easier for users doing things interactively

So a CLI will we have to "take" for all specs, where a spec config only "takes" on the specific spec. We can consider adding it but I don't think we can similar flags (that override spec configuration(s))

Copy link
Copy Markdown
Contributor

@yevgenypats yevgenypats left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. One nit:

Can we move migrate enum code to migrate.go as looks like our enums are growing and maybe also add a small sanity test for marshal/unmarsahl.

@erezrokah erezrokah force-pushed the feat/add_migrate_mode branch from 6ccda38 to 1ff6062 Compare January 15, 2023 16:22
@erezrokah
Copy link
Copy Markdown
Member Author

Can we move migrate enum code to migrate.go as looks like our enums are growing and maybe also add a small sanity test for marshal/unmarsahl.

Sure, good point. Let me know if 1ff6062 is what you had in mind

@erezrokah erezrokah merged commit 78b9acb into cloudquery:main Jan 15, 2023
kodiakhq bot pushed a commit that referenced this pull request Jan 16, 2023
🤖 I have created a release *beep* *boop*
---


## [1.26.0](v1.25.1...v1.26.0) (2023-01-16)


### Features

* **destinations:** Add `migrate_mode` ([#604](#604)) ([78b9acb](78b9acb))


### Bug Fixes

* **destination:** Pass proper spec to client constructor ([#606](#606)) ([8370882](8370882))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
kodiakhq bot pushed a commit to cloudquery/cloudquery that referenced this pull request Jan 19, 2023

#### Summary

To be merged after #6759, and needs cloudquery/plugin-sdk#604

<!--
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants