feat: Add sharding support#19169
Conversation
cli/cmd/sync_v3.go
Outdated
| syncTime := time.Now().UTC().Truncate(time.Microsecond) | ||
| sourceName := sourceSpec.Name | ||
| if shard != nil { | ||
| sourceName = fmt.Sprintf("%s-%d/%d", sourceName, shard.num, shard.total) |
There was a problem hiding this comment.
This is needed to support overwrite-delete-stale otherwise shards delete each other's data at the end of the sync.
We can use a better name for it, maybe %s_shard_%d_%d
4e6767b to
608a269
Compare
cli/cmd/sync.go
Outdated
| cmd.Flags().String("license", "", "set offline license file") | ||
| cmd.Flags().String("summary-location", "", "Sync summary file location. This feature is in Preview. Please provide feedback to help us improve it.") | ||
| cmd.Flags().String("tables-metrics-location", "", "Tables metrics file location. This feature is in Preview. Please provide feedback to help us improve it. Works with plugins released on 2024-07-10 or later.") | ||
| cmd.Flags().String("shard", "", "Allows splitting the sync process into multiple shards. This feature is in Preview. Please provide feedback to help us improve it. Works with plugins released on 2024-09-24 or later.") |
There was a problem hiding this comment.
This does not mention that it only works for plugins using the default scheduler (e.g. not S3), but I realise it's a little tricky to put this in words.
There was a problem hiding this comment.
Yeah, it doesn't work with Docker plugins as well 🙃
I think that will be in the update of https://docs.cloudquery.io/docs/advanced-topics/running-cloudquery-in-parallel and https://docs.cloudquery.io/docs/deployment/github-actions
There was a problem hiding this comment.
OK so did fff4d1d, I'll update the doc with a table of support plugins and versions
|
I'll merge this once we have at least AWS, GCP and Azure published with this support |
fff4d1d to
e3dd9af
Compare
🤖 I have created a release *beep* *boop* --- ## [6.8.0](cli-v6.7.1...cli-v6.8.0) (2024-09-19) ### Features * Add sharding support ([#19169](#19169)) ([e9dfd0b](e9dfd0b)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Summary
We'll need to release a few plugins with cloudquery/plugin-sdk#1891 first, hence the future date in the command description