feat!: Add DeleteRecord handling to Clickhouse destination#20772
feat!: Add DeleteRecord handling to Clickhouse destination#20772kodiakhq[bot] merged 16 commits intomainfrom
Conversation
|
🚀 ClickHouse Cloud UI deployed to Vercel: |
|
🚀 PostgreSQL Cloud UI deployed to Vercel: |
| return err | ||
| } | ||
|
|
||
| if err = c.conn.Exec(ctx, sql, params...); err != nil { |
There was a problem hiding this comment.
I'd like to be able to use some kind of prepared statement or batching here, but CH does not support this for DELETE statements. Batching is only supported for INSERT queries.
| services: | ||
| clickhouse: | ||
| image: clickhouse/clickhouse-server:22.1.2 | ||
| image: clickhouse/clickhouse-server:24.12.1 |
There was a problem hiding this comment.
Is this needed to support deletion? If so we should do a major version bump and update the docs https://hub.cloudquery.io/plugins/destination/cloudquery/clickhouse/latest/docs#overview
There was a problem hiding this comment.
Yes that's to support the lightweight delete statements. Those were added in CH 23.3 (release notes). The current latest version of CH is 25.4.4. I'm wondering what we should set as the new minimum version for this plugin. WDYT? 23.3?
There was a problem hiding this comment.
Per https://clickhouse.com/docs/whats-new/changelog/2023 23.3 was released over 2 years ago and no longer supported (LTS are supported for 1 year per https://clickhouse.com/docs/knowledgebase/production#how-to-choose-between-clickhouse-releases).
Maybe we can move to 24.8 LTS which is the latest LTS which is still supported https://clickhouse.com/docs/whats-new/changelog/2024#a-id248a-clickhouse-release-248-lts-2024-08-20
There was a problem hiding this comment.
Maybe even 25.3 as 24.8 will go out of support soon, and 25.3 has JSON type production level support
There was a problem hiding this comment.
Regardless needs to be a breaking change
There was a problem hiding this comment.
Yeah it does need to be breaking, agreed. I think jumping to 25.x will be a problem for our platform tenants. On platform-preview we run version 24.12.1 of CH. I think I'll bump the plugin to 24.8.1.
🤖 I have created a release *beep* *boop* --- ## [7.0.0](plugins-destination-clickhouse-v6.2.7...plugins-destination-clickhouse-v7.0.0) (2025-05-20) ### ⚠ BREAKING CHANGES * Add DeleteRecord handling to Clickhouse destination ([#20772](#20772)) ### Features * Add DeleteRecord handling to Clickhouse destination ([#20772](#20772)) ([13e9573](13e9573)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.80.2 ([#20778](#20778)) ([525352c](525352c)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.80.3 ([#20802](#20802)) ([2ba2f8e](2ba2f8e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
A small oversight in #20772.
🤖 I have created a release *beep* *boop* --- ## [9.0.0](plugins-destination-postgresql-v8.8.7...plugins-destination-postgresql-v9.0.0) (2025-05-21) ### ⚠ BREAKING CHANGES * Add DeleteRecord handling to Clickhouse destination ([#20772](#20772)) ### Features * Add DeleteRecord handling to Clickhouse destination ([#20772](#20772)) ([13e9573](13e9573)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.80.2 ([#20778](#20778)) ([525352c](525352c)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.80.3 ([#20802](#20802)) ([2ba2f8e](2ba2f8e)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.81.0 ([#20806](#20806)) ([567e252](567e252)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This adds handling for
DeleteRecordmessages in ClickHouse along with tests. I've also enabled theDeleteRecordtests in Postgres as they were excluded by mistake.BEGIN_COMMIT_OVERRIDE
chore: Add DeleteRecord handling to Clickhouse destination
END_COMMIT_OVERRIDE