fix(main): Write to mysql in bulk#19123
Merged
erezrokah merged 6 commits intocloudquery:mainfrom Sep 16, 2024
Merged
Conversation
added 2 commits
September 7, 2024 00:14
Contributor
|
Hey @jsifuentes , Thanks so much for submitting this PR! There might some short delay for the review here as our team is at an offsite at the moment. |
erezrokah
approved these changes
Sep 16, 2024
Member
erezrokah
left a comment
There was a problem hiding this comment.
Thanks for the PR @jsifuentes looks great. Sorry for the late review, I wanted to make sure we're not hitting query size limits. Looks like our batching limits prevent that from happening (unless someone increases them simnifically).
I also added a test to see, on my local machine the improvement of the test time was fro m 127.894s to 39.591s which is a great 🚀
kodiakhq bot
pushed a commit
that referenced
this pull request
Sep 16, 2024
🤖 I have created a release *beep* *boop* --- ## [5.2.8](plugins-destination-mysql-v5.2.7...plugins-destination-mysql-v5.2.8) (2024-09-16) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.62.0 ([#19137](#19137)) ([ed315d0](ed315d0)) * **main:** Write to mysql in bulk ([#19123](#19123)) ([0d1b9d5](0d1b9d5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Member
|
Released in https://github.com/cloudquery/cloudquery/releases/tag/plugins-destination-mysql-v5.2.8, thanks again @jsifuentes 💯 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #19122
Before, the
INSERTquery was built within theappendTableBatchandoverwriteTableBatchfunctions. But we aren't yet aware of how many rows need to be inserted, so we don't know how many value blocks to add. Rows are iterated withinwriteResourcesto create the final records to insert, so this is an opportunity to calculate a row count. So I moved the query building to withinwriteResourcesand constructed theINSERTquery based on the amount of rows to be inserted.This reduced the time to write 27,000 records from 20 minutes to under 1 minute.