-
Notifications
You must be signed in to change notification settings - Fork 8.4k
Size-capped tables #37548
Copy link
Copy link
Open
Labels
featurewarmup taskThe task for new ClickHouse team members. Low risk, moderate complexity, no urgency.The task for new ClickHouse team members. Low risk, moderate complexity, no urgency.
Metadata
Metadata
Assignees
Labels
featurewarmup taskThe task for new ClickHouse team members. Low risk, moderate complexity, no urgency.The task for new ClickHouse team members. Low risk, moderate complexity, no urgency.
Type
Fields
Give feedbackNo fields configured for issues without a type.
Use case
Logs and events.
Describe the solution you'd like
Four new table-level settings for MergeTree tables:
The oldest parts (by insertion time, a.k.a min block number) will be deleted if the total table size is more than
max_..._to_keepand will be not less thanmin_..._to_keepafter deletion.Caveats
We cut the table not precisely by size but only by dropping the oldest parts, as it is cheap.
For large tables it may lead to a variation of up to 150 GB in size in worst case.
Can be solved with some tuning (maybe even automatic) of max parts size to merge.
Cutting off the excessive records can also be done on merging of the oldest parts. It can be done precisely for the number of rows and by proration for the number of bytes.
Additional Context
This is for simple cases by removing the oldest inserted data.
Note that for removing the data by some values (e.g. datetime field), and even for out-of-order data, is already possible with TTL.