-
Notifications
You must be signed in to change notification settings - Fork 555
db: add external sstable iterator interface #1526
Description
Clients, including CockroachDB, sometimes have a need to iterate over the merged keys of several external sstables. In CockroachDB, backups are written in an sstable format. Incremental backups record a diff of changed state. During restore, CockroachDB must iterate over the backup sstables to reconstruct the current state, merging across sstables. Today, this is performed by pkg/storage.multiIterator.
We should provide a facility for external users like CockroachDB to use Pebble's own internal iterators, especially now that iteration has become more complicated with interleaving range keys.
One implementation idea is to allow constructing *pebble.Iterator that reads these external sstables through constructing an artificial readState representing the input set of sstables. We could require that the sstables have all-zero sequence numbers, and assign each of sstable within the input set an artificial global sequence number, so that invariant checks and zero-seqnum optimizations aren't run afoul.