-
Notifications
You must be signed in to change notification settings - Fork 555
compaction: integrate range keys #1686
Copy link
Copy link
Closed
Description
Range keys (#1339) need to be read and written during compactions. Like range deletions, range keys exist separately from point keys and are surfaced through the keyspan.FragmentIterator interface. Here's a sketch of the anticipated integration.
- When constructing a compaction input iterator, a
keyspan.LevelIteris created for each input level that contains at least 1 file with range keys. - The individual levels' range key
keyspan.LevelIters are wrapped in akeyspan.MergingIter. TheMergingIteris configured with a special keyspan.Transform. TheTransformis configured with the slice of open snapshots. It partitions aSpan'sKeysinto snapshot stripes, calling rangekey.Coalesce on theKeysof each stripe. We may want to refactorrangekey.Coalesce's interface to only take an input[]Keysslice and output*[]Keyspointer. Therangekey.Coalescecalls performs shadowing, dropping unnecessary keys. ThisTransformis also likely where we should examine in-use key ranges (eg,elideRangeTombstone) to determine whether or not to dropRangeKeyUnsetandRangeKeyDeletekeys that exist in the last snapshot stripe. - The
MergingIteris then wrapped in akeyspan.DefragmentingIterconfigured with theDefragmentInternaldefragmenting method. The defragmenting iterator will ensure the fragments persisted to disk are only as fine-grained as required. - Finally, the
keyspan.DefragmentingIteris wrapped with akeyspan.InterleavingIterto interleave range keys into the main compaction loop. This is similar to what was done for range deletions in compaction: use InterleavingIter to interleave rangedels #1659. - When the main compaction loop encounters a
RANGEKEYSET,RANGEKEYUNSETorRANGEKEYDELETE, it retrieves the span from the interleaving iterator and adds it to akeyspan.Fragmenter(separate fromc.rangeDelFrag). All the truncation logic mirrors range deletions. Range keys should be added to the output sstable using Writer.AddRangeKey (not theWriter.RangeKey{Set,Unset,Delete}methods). Future work may consider removing thekeyspan.Fragmenter, but I think it should be deferred for now, and we should track as close to possible to the current handling of range deletions.
Reactions are currently unavailable