Relates to #50194.

Garbage collection of MVCC versions is currently excessively disruptive to foreground traffic on the keys being garbage collected. This is because GCRequests acquire write latches over each of the keys being GCed. The idea here is to use latches to protect GC requests from interacting poorly with reads near the GC threshold and to protect them from interacting poorly with other GC requests on the same key (e.g. to avoid duplicate stats changes).

The problem is that both of these uses of latches are broken. The first use is broken because we acquire write latches at the batch header's timestamp, which is set to time.Now(), so we're only serializing with reads in the future and all other writes [1]. So we're disruptive to everyone except who we want to serialize with – reads in the past! The second use is broken because we can already serialize with other GCRequests on a separate key, such as the RangeLastGCKey. So we end up going out of our way to cause GC requests to disrupt foreground traffic and don't get anything from it.

To address these issues, I propose that we make garbage collection latch-less. GC requests should no longer acquire latches on the keys that they intend to GC. Instead, they should only grab a latch on the RangeLastGCKey. To ensure that other requests properly synchronize with GC requests, they will now need to check the GC threshold after evaluation as well [2]. I've mocked this out in this branch: nvb@8486020.

[1] I think it's fine for GC requests on old versions of a key to commute with writes to new versions of a key in terms of their stats updates, but we'll need to confirm that.

[2] This could be simplified if we made Engine.NewReadOnly grab an engine snapshot so that evaluation would always work on a stable view of the MVCC keyspace. In that world, we could simply check the GC threshold after calling Engine.NewReadOnly on the read request path.

Jira issue: CRDB-3679