We currently throttle when adding sstables to the LSM due to background bulk operations. This throttling happens in two places, in Store.Send, prior to the proposal, and in addSSTablePreApply, when applying to the state machine at each replica. The latter is problematic in that it (a) delays applying later things in the raft log at the replica, and (b) blocks a worker on the raft scheduler, which can impact other ranges.

The Store.Send throttling only looks at the store health on the leaseholder, and we have seen occurrences where one node (or a few nodes) in a cluster has an unhealthy LSM store. If we maintained soft state in a node about the health of other stores in the system (this state would be slow changing), we could use it in Store.Send and remove the apply-time throttling.

This store health information can also be useful for throttling in the GC queue.

Jira issue: CRDB-2847

Epic CRDB-15069