Skip to content

fix: Add more verbose error for when partition and sorting key changes#21265

Merged
kodiakhq[bot] merged 3 commits intomainfrom
feature/eng-1945-improve-error-message-when-migrating-from-clickhouse-4x-to
Aug 20, 2025
Merged

fix: Add more verbose error for when partition and sorting key changes#21265
kodiakhq[bot] merged 3 commits intomainfrom
feature/eng-1945-improve-error-message-when-migrating-from-clickhouse-4x-to

Conversation

@hermanschaaf
Copy link
Copy Markdown
Contributor

@hermanschaaf hermanschaaf commented Aug 19, 2025

This improves the error message when migrating between plugin versions where sorting key or partition keys have changed. Before, the user was left guessing what changes were needed for certain tables.

Before

cloudquery sync aws_to_clickhouse.yaml
Loading spec(s) from aws_to_clickhouse.yaml
Starting sync for: aws (cloudquery/aws@v32.37.1) -> [clickhouse (cloudquery/clickhouse@v7.3.1)]
Error: failed to sync v3 source aws: write client returned error (insert): plugin returned error:
Can't migrate tables automatically, migrate manually or consider using 'migrate_mode: forced'. Non auto migratable tables changes:

aws_ec2_instance_types:
  - Column "placement_group_info" added with type "utf8"
  - Column "test_column" with type "utf8" removed
  - Not null constraint removed from column "v_cpu_info"

aws_eks_cluster_node_groups:


aws_elasticache_replication_groups:


aws_elbv1_load_balancers:


aws_elbv2_load_balancers:


aws_rds_clusters:


aws_rds_instances:

After:

cloudquery sync aws_to_clickhouse.yaml
Loading spec(s) from aws_to_clickhouse.yaml
Starting sync for: aws (cloudquery/aws@v32.37.1) -> [clickhouse (local@/Users/herman/code/cloudquery/cloudquery/plugins/destination/clickhouse/clickhouse)]
Error: failed to sync v3 source aws: write client returned error (insert): plugin returned error:

Can't migrate tables automatically, migrate manually or consider using 'migrate_mode: forced'.

Non auto-migratable tables:

aws_ec2_instance_types:
  - Column "placement_group_info" added with type "utf8"
  - Column "test_column" with type "utf8" removed
  - Not null constraint removed from column "v_cpu_info"
  - Sorting key changed (was [supported_boot_modes,supported_root_device_types,supported_usage_classes,supported_virtualization_types,_cq_id] and would become [_cq_id])

aws_eks_cluster_node_groups:
  - Sorting key changed (was [instance_types,subnets,_cq_id] and would become [_cq_id])

aws_elasticache_replication_groups:
  - Sorting key changed (was [member_clusters,member_clusters_outpost_arns,user_group_ids,_cq_id] and would become [_cq_id])

aws_elbv1_load_balancers:
  - Sorting key changed (was [availability_zones,security_groups,subnets,_cq_id] and would become [_cq_id])

aws_elbv2_load_balancers:
  - Sorting key changed (was [security_groups,_cq_id] and would become [_cq_id])

aws_rds_clusters:
  - Sorting key changed (was [availability_zones,custom_endpoints,enabled_cloudwatch_logs_exports,read_replica_identifiers,_cq_id] and would become [_cq_id])

aws_rds_instances:
  - Sorting key changed (was [enabled_cloudwatch_logs_exports,read_replica_db_cluster_identifiers,read_replica_db_instance_identifiers,_cq_id] and would become [_cq_id])

@hermanschaaf hermanschaaf changed the title fix: Add more verbose error for when partition and sorting key changes fix(clickhouse): Add more verbose error for when partition and sorting key changes Aug 19, 2025
@cq-bot
Copy link
Copy Markdown
Contributor

cq-bot commented Aug 19, 2025

@hermanschaaf hermanschaaf changed the title fix(clickhouse): Add more verbose error for when partition and sorting key changes fix: Add more verbose error for when partition and sorting key changes Aug 19, 2025
@hermanschaaf hermanschaaf marked this pull request as ready for review August 20, 2025 07:41
@hermanschaaf hermanschaaf requested a review from a team as a code owner August 20, 2025 07:41
@hermanschaaf hermanschaaf requested review from dcelasun and marianogappa and removed request for dcelasun August 20, 2025 07:41
@hermanschaaf hermanschaaf added the automerge Automatically merge once required checks pass label Aug 20, 2025
@kodiakhq kodiakhq bot merged commit 52d3e7f into main Aug 20, 2025
21 checks passed
@kodiakhq kodiakhq bot deleted the feature/eng-1945-improve-error-message-when-migrating-from-clickhouse-4x-to branch August 20, 2025 08:16
kodiakhq bot pushed a commit that referenced this pull request Aug 27, 2025
🤖 I have created a release *beep* *boop*
---


## [7.3.2](plugins-destination-clickhouse-v7.3.1...plugins-destination-clickhouse-v7.3.2) (2025-08-27)


### Bug Fixes

* Add more verbose error for when partition and sorting key changes ([#21265](#21265)) ([52d3e7f](52d3e7f))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.89.0 ([#21238](#21238)) ([deb4cee](deb4cee))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/plugin/destination/clickhouse automerge Automatically merge once required checks pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants