importccl: compute expected number of data columns only once#55846
Closed
pbardea wants to merge 2 commits intocockroachdb:masterfrom
Closed
importccl: compute expected number of data columns only once#55846pbardea wants to merge 2 commits intocockroachdb:masterfrom
pbardea wants to merge 2 commits intocockroachdb:masterfrom
Conversation
Member
|
This change is |
Contributor
Author
|
First commit is #55845. As expected, memory usage of Row() goes down quite a bit. Will follow up with profiles, but it might be nice to also get a sense of any impact on import performance. |
32a4b91 to
e240695
Compare
The current implementation of checking for validating the number of rows for computed columns for in non-IMPORT INTO backups is inefficient. This commit moves the check from being performed on every row to only being performed once per import. Release note: None
This commit moves the computation of finding the expected number of data columns to the creation of the input converter, rather than re-calculating it on every row. The memory footprint of loading all of the visible columns for every row was noticeable. Release note (performance improvement): CSV imports should now be slightly faster.
e240695 to
007bc36
Compare
Contributor
Author
|
Closing in favour of folding this into #55846. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit moves the computation of finding the expected number of data
columns to the creation of the input converter, rather than
re-calculating it on every row. The memory footprint of loading all of
the visible columns for every row was noticeable.
Release note (performance improvement): CSV imports should now be
slightly faster.