Skip to content

db: add external sstable iterator interface #1526

@jbowens

Description

@jbowens

Clients, including CockroachDB, sometimes have a need to iterate over the merged keys of several external sstables. In CockroachDB, backups are written in an sstable format. Incremental backups record a diff of changed state. During restore, CockroachDB must iterate over the backup sstables to reconstruct the current state, merging across sstables. Today, this is performed by pkg/storage.multiIterator.

We should provide a facility for external users like CockroachDB to use Pebble's own internal iterators, especially now that iteration has become more complicated with interleaving range keys.


One implementation idea is to allow constructing *pebble.Iterator that reads these external sstables through constructing an artificial readState representing the input set of sstables. We could require that the sstables have all-zero sequence numbers, and assign each of sstable within the input set an artificial global sequence number, so that invariant checks and zero-seqnum optimizations aren't run afoul.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions