feat: Implement PARTITION BY support on ClickHouse destination.#19596
Merged
kodiakhq[bot] merged 2 commits intomainfrom Nov 13, 2024
Merged
feat: Implement PARTITION BY support on ClickHouse destination.#19596kodiakhq[bot] merged 2 commits intomainfrom
kodiakhq[bot] merged 2 commits intomainfrom
Conversation
Contributor
|
🚀 ClickHouse Cloud UI deployed to Vercel: |
erezrokah
approved these changes
Nov 13, 2024
Member
erezrokah
left a comment
There was a problem hiding this comment.
Looks good 👍 Makes sense to have it as a passthrough string and not a list of columns
disq
approved these changes
Nov 13, 2024
Contributor
Author
kodiakhq bot
pushed a commit
that referenced
this pull request
Nov 13, 2024
🤖 I have created a release *beep* *boop* --- ## [5.1.0](plugins-destination-clickhouse-v5.0.9...plugins-destination-clickhouse-v5.1.0) (2024-11-13) ### Features * Implement PARTITION BY support on ClickHouse destination. ([#19596](#19596)) ([503f42a](503f42a)) ### Bug Fixes * **deps:** Update dependency @types/jest to v29.5.14 ([#19544](#19544)) ([f0340e5](f0340e5)) * **deps:** Update dependency @types/node to v16.18.119 ([#19545](#19545)) ([299926d](299926d)) * **deps:** Update material-ui monorepo ([#19548](#19548)) ([c3f765e](c3f765e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Enables optional support for partitioning tables in ClickHouse destination.
Because this came up in discussions as a potential eyebrow raiser, note that I decided to expose the partitioning as a pass-through string, rather than a list of columns, which was suggested:
toYYYYMM(_cq_sync_time). So partitioning by column names would make this feature unusable for many use cases, and also I wouldn't want us to need to implement a semantic validation of all possible expressions that can be used here.Docs:
Partitioning
This option allows to specify a partitioning strategy to be used for tables. It is an array of objects.
Each object has the following fields:
tables(array of strings) (optional) (default:["*"])List of glob patterns to match table names against. Follows the same rules as the top-level spec
tablesoption.If a table matches both a pattern in
tablesandskip_tables, the table will be skipped.Partition strategy table patterns should be disjointed sets: if a table matches two partition strategies, an error will be raised at runtime.
skip_tables(array of strings) (optional) (default: empty)List of glob patterns to skip matching table names against. Follows the same rules as the top-level spec
skip_tablesoption.If a table matches both a pattern in
tablesandskip_tables, the table will be skipped.Partition strategy table patterns should be disjointed sets: if a table matches two partition strategies, an error will be raised at runtime.
partition_by(string) (required)Partitioning strategy to use, e.g.
toYYYYMM(_cq_sync_time), the string is passed as is after "PARTITION BY" clause with no validation or quoting.An unset
partition_byis not valid.Example: