feat: Add Snowflake destination plugin#4965
Merged
kodiakhq[bot] merged 36 commits intomainfrom Nov 24, 2022
Merged
Conversation
disq
reviewed
Nov 23, 2022
disq
requested changes
Nov 24, 2022
Co-authored-by: Kemal <223029+disq@users.noreply.github.com>
Co-authored-by: Kemal <223029+disq@users.noreply.github.com>
disq
requested changes
Nov 24, 2022
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
hermanschaaf
suggested changes
Nov 24, 2022
Contributor
hermanschaaf
left a comment
There was a problem hiding this comment.
Looks good! Just a couple of comments from a quick read through
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
disq
reviewed
Nov 24, 2022
disq
reviewed
Nov 24, 2022
disq
reviewed
Nov 24, 2022
Co-authored-by: Kemal <223029+disq@users.noreply.github.com>
Co-authored-by: Kemal <223029+disq@users.noreply.github.com>
disq
reviewed
Nov 24, 2022
Co-authored-by: Kemal <223029+disq@users.noreply.github.com>
Co-authored-by: Kemal <223029+disq@users.noreply.github.com>
disq
reviewed
Nov 24, 2022
hermanschaaf
approved these changes
Nov 24, 2022
disq
approved these changes
Nov 24, 2022
kodiakhq bot
pushed a commit
that referenced
this pull request
Nov 24, 2022
🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2022-11-24) ### Features * Add Snowflake destination plugin ([#4965](#4965)) ([9650bcb](9650bcb)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a snowflake destination plugin.
This support streaming directly to snowflake (via local csv uploads as streaming insert is not supporting all snowflake data-types as of right now).
The streaming way (or psudo streaming) is useful to get started and to test stuff locally but for production usages it is recommended to use standard data-pipelines that uploads first csv/json to a remote storage (S3, GCS, Azure Blob...) and then via a periodic job or snowpipe to load it to Snowflake DB will be the most performant and cheap way to do at scale.
For the latter, this PR needs to be implemented and tested #4902
Known Issue: Migrations are slow due to currently not using any batching to get all the tables. I've an idea on how to solve it but maybe will do it in a follow-up PR.