In 21.2 we taught the backup processor to stitch smaller SST files returned as part of the ExportResponse into larger SST files before shipping them off to the ExternalStorage bucket. This results in later incremental backups in the chain, having to unmarshal and buffer far fewer files from previous backups, and consequently reduces the memory footprint of backup.

As part of this work, we added a few configurable settings that governed the size of the buffer that we used when stitching and the target file size we attempted to hit before flushing the file. Recent experiments have shown that the default values of these cluster settings might be suboptimal, and with a little tuning we can reap more rewards. This is a tracking issue for all the investigative work, and future fixes that will stem from the investigation.

Epic CRDB-10260