Make read buffer size lower, while reading from compact parts#12492
Make read buffer size lower, while reading from compact parts#12492CurtizJ merged 5 commits intoClickHouse:masterfrom
Conversation
13de710 to
1a11bc7
Compare
c6423b2 to
6963288
Compare
There was a problem hiding this comment.
I think I've got the idea: we need for find the largest granule and limit buffer size if possible.
Still, comment will be helpful: what this function does and what why do we care calculating this stuff instead of using max_read_buffer_size.
Also, it is not clear what would we use inside function. Like we need column_positions filled, data_part, marks_loader (is it all?). This function may be made static with implicit arguments to avoid possible usage of uninitialized members.
There was a problem hiding this comment.
It would be really nice to implement iterator here. Cause this lambda captures all variables, and some of them changed later in unpredictable way. Readability of it is pretty low.
KochetovNicolai
left a comment
There was a problem hiding this comment.
It would be nice to make a perftest which shows that something has improved.
6963288 to
61018c2
Compare
|
I also added one change from #12183 to this PR. Now compressed block is written for every part of column in granule. It should significanty increase performance of reading, but maybe writing will slow down. Let's look to perf tests. |
3998192 to
d3fbeb2
Compare
And if it's Ok let's apply this change unconditionally (without a setting). |
|
Perf tests look ok. Insert: Select: |
Backport #12492 to 20.6: Make read buffer size lower, while reading from compact parts
Backport #12492 to 20.4: Make read buffer size lower, while reading from compact parts
Backport #12492 to 20.5: Make read buffer size lower, while reading from compact parts
Backport #12492 to 20.3: Make read buffer size lower, while reading from compact parts
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixed performance issue, while reading from compact parts.