Reduce INSERT/merges memory usage with wide parts for very wide tables by enabling adaptive write buffers#92250
Merged
azat merged 2 commits intoClickHouse:masterfrom Dec 18, 2025
Conversation
Contributor
|
Workflow [PR], commit [c64ea66] Summary: ❌
|
Avogar
approved these changes
Dec 16, 2025
alexey-milovidov
approved these changes
Dec 16, 2025
c215488 to
82fca78
Compare
…aptive write buffers Each column requires at least 1MiB buffer for writing, and in case you have 1000 columns you need at least 1GiB to write part (on INSERT or merge). Introduce new MergeTree setting `min_columns_to_activate_adaptive_write_buffer`, that controls when to use adaptive write buffers, techinque that uses adaptive buffers for writing, that starts from 16K For metric_log memory usage decreased 30x for INSERTing/merging (horizontally) wide parts (note, that vertical merge may be significantly slower in case of tons of columns). Also note, that default value (500) automatically enables it for system.metric_log, and applies this optimization for it.
82fca78 to
c64ea66
Compare
Member
Author
|
@Avogar please take a look one more time, I've touched encrypted disk, though it should pretty straightforward, I could use one more pair of eyes |
Merged
via the queue into
ClickHouse:master
with commit Dec 18, 2025
4f2f026
127 of 131 checks passed
This was referenced Dec 18, 2025
Closed
zvonand
pushed a commit
to Altinity/ClickHouse
that referenced
this pull request
Dec 29, 2025
…_adaptive_write_buffer Reduce INSERT/merges memory usage with wide parts for very wide tables by enabling adaptive write buffers
25 tasks
1 task
zvonand
pushed a commit
to Altinity/ClickHouse
that referenced
this pull request
Jan 15, 2026
…_adaptive_write_buffer Reduce INSERT/merges memory usage with wide parts for very wide tables by enabling adaptive write buffers
25 tasks
zvonand
pushed a commit
to Altinity/ClickHouse
that referenced
this pull request
Jan 27, 2026
…_adaptive_write_buffer Reduce INSERT/merges memory usage with wide parts for very wide tables by enabling adaptive write buffers
25 tasks
zvonand
added a commit
to Altinity/ClickHouse
that referenced
this pull request
Feb 3, 2026
25.8.15 Backport of ClickHouse#92250 - Reduce INSERT/merges memory usage with wide parts for very wide tables by enabling adaptive write buffers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Reduce INSERT/merges memory usage with wide parts for very wide tables by enabling adaptive write buffers. Add support of adaptive write buffers for encrypted disks.
Each column requires at least 1MiB buffer for writing, and in case you have 1000 columns you need at least 1GiB to write part (on INSERT or merge).
Introduce new MergeTree setting
min_columns_to_activate_adaptive_write_buffer, that controls when to use adaptive write buffers, techinque that uses adaptive buffers for writing, that starts from 16KFor metric_log memory usage decreased 30x for INSERTing/merging (horizontally) wide parts (note, that vertical merge may be significantly slower in case of tons of columns).
Also note, that default value (500) automatically enables it for system.metric_log, and applies this optimization for it.