-
Notifications
You must be signed in to change notification settings - Fork 409
Closed
Labels
affects-6.4component/storageseverity/majortype/enhancementThe issue or PR belongs to an enhancement.The issue or PR belongs to an enhancement.
Description
Enhancement
Suppose we write a ColumnFileBig into the memtable of a segment without any delta:
-Inf +Inf
|<------------------------------>| Segmemt
|<-CFBig->|
Then physical split happens:
-Inf +Inf
|<--------------->|<------------>| Segmemt
|<-CFBig->|
The ColumnFileBig is now referenced by both two segments. And,
-
The right segment may trigger a delta merge because the delta layer is big.
-
The left segment will not trigger a delta merge because the delta is still empty -- Its referenced CFBig is not contained in the segment.
As a result, the ColumnFileBig is kept being referenced and not recycled, until user manually triggers a DeltaMerge for all segments.
This happens when we ingest SSTs quickly (using a higher ingest concurrency), result in 25% space amplification in my experiment.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
affects-6.4component/storageseverity/majortype/enhancementThe issue or PR belongs to an enhancement.The issue or PR belongs to an enhancement.