In internal testing of Pebble on one node of an internal test cluster, we noticed increased read ops on the Pebble node. (Pebble is n3, the node shown in red).

Further investigation is pointing the finger at the read op increase occurring during backups. Incremental backups when Pebble was enabled on n3 were taking ~40min. After switching n3 back to RocksDB the next incremental backup took 7m12s.

Nightly backup roachtests which perform full backups do not show any time difference between RocksDB and Pebble.

It is possible there is a big in the timebound iterator code, or in the ExportToSst code (which lies at the heart of backup) which is causing significantly increased reads that in turn leads to slower backups. @dt will be doing some experimentation on another test cluster. @joshimhoff is pulling the sizes of recent Pebble and RocksDB generated incremental backups to see if they are similar in size.

This is a developing story and will be updated soon.