feat(sqlite): Collect and report migration errors before starting the migration#6759
Merged
kodiakhq[bot] merged 4 commits intocloudquery:mainfrom Jan 19, 2023
Conversation
erezrokah
commented
Jan 12, 2023
| switch { | ||
| case sqliteColumn == nil: | ||
| c.logger.Debug().Str("table", table.Name).Str("column", col.Name).Msg("Column doesn't exist, creating") | ||
| sql := "alter table \"" + table.Name + "\" add column \"" + columnName + "\" \"" + columnType + `"` |
Member
Author
There was a problem hiding this comment.
We only modify table columns when we add a new one and it's not a PK. I kept that logic
erezrokah
commented
Jan 12, 2023
| if info, err = c.getTableInfo(table.Name); err != nil { | ||
| return fmt.Errorf("failed to get table %s columns types: %w", table.Name, err) | ||
| migrationErrors := getMigrationErrors(schemaChanges) | ||
| if len(migrationErrors) > 0 { |
Member
Author
There was a problem hiding this comment.
When we add the --force flag, we'll log the errors and continue (the code to handle the errors, e.g. dropping a table is not here yet, I'll add that when I add the --force flag)
fa4d5b5 to
0f3ab0a
Compare
f8b8186 to
222f60a
Compare
67e1cb4 to
e1119f4
Compare
hermanschaaf
approved these changes
Jan 19, 2023
Member
Author
|
Thanks for the review @hermanschaaf. I'll merge this as it's blocking another PR cc @yevgenypats if you have concerns I can follow up |
kodiakhq bot
pushed a commit
that referenced
this pull request
Jan 19, 2023
#### Summary To be merged after #6759, and needs cloudquery/plugin-sdk#604 <!--
kodiakhq bot
pushed a commit
that referenced
this pull request
Jan 24, 2023
#### Summary Related to #6600, #6763 and #6759 This PR adds support for PK changes migration. Before we would just ignore them, so the following use cases were ignored: 1. Adding a new column as a primary key -> The column was added, but the table PKs were left unchanged 2. Removing a PK from an existing column -> Was ignored 3. Adding a PK to an existing column -> Was ignored 4. Changing the type of an existing PK column -> Was handled by a regular non PK column type change. We dropped the column, but we need to drop the table. SQLite is quite limited in what can be done with PK changes. On any PK change we need to recreate the table, so either `drop + create` or `create new + copy old to new + drop old + rename new`. I went with the former for simplicity. <!--
kodiakhq bot
pushed a commit
that referenced
this pull request
Jan 24, 2023
🤖 I have created a release *beep* *boop* --- ## [1.2.0](plugins-destination-sqlite-v1.1.6...plugins-destination-sqlite-v1.2.0) (2023-01-24) ### Features * **sqlite-migrate:** Support PK changes to schema ([#7006](#7006)) ([dddd852](dddd852)) * **sqlite:** Collect and report migration errors before starting the migration ([#6759](#6759)) ([a80e9d9](a80e9d9)) * **sqlite:** Support force migration ([#6763](#6763)) ([19bba77](19bba77)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.28.0 ([#7009](#7009)) ([12ac005](12ac005)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related to #6600.
Still in draft as I need to test it 😄At the moment we migrate the tables and columns one by one and stop if we can't migrate due to potential data loss.
This has a few issues
Notes
--forcethe change will be simply to ignore the errors and keep on migrating (e.g. drop/add columns/tables)