Skip to content

backupccl, engine: re-enable Time-Bound Iterator in ExportToSST #43799

@dt

Description

@dt

TBI looks like it could be a big deal in getting to low-RPO via frequent incremental backups and indeed, is potentially even required to get there. On paper it sounds like the perfect solution, so what do we need to do to regain enough confidence in it and re-enable its use in BACKUP?

Context refresher: Within the current BACKUP/RESTORE feature offering, the way to reduce RPO is to backup more often. For this to work for more frequent durations, the cost of backing up what changed would scale with the size what changed as opposed to the total data size. This is exactly what the time-bound iterator optimization is supposed to deliver -- by recording the span of timestamps that appear in any given SSTable and then installing a filter to only open sstables that contain relevant times, we can ignore irrelevant data very cheaply.

However over 2017 and 2018, we encountered a few tricky correctness issues in the interaction of the time-bound iterator and our MVCC logic and handling intents, and generally lost confidence in this code, ultimately disabling its use in the scans used by BACKUP (though, as far as i know, we don't currently actually know of any remaining bugs in it, right?)
At the time, we did some tests that indicated that on a freshly made and populated cluster, backing up a day's worth of changes, it didn't have a particularly large impact, and was thus not enough to outweigh our general acquired distrust of it even thouhg we'd fixed all the known bugs.
However more recently we've heard from customers interested in RPO on the scale of a minute rather than a day, running in clusters that contain terabytes of data. Without TBI, generating incremental such backups looks very expensive -- potentially prohibitively so.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions