Is your feature request related to a problem? Please describe.

We've seen large backlogs of GC-able keys build up when installing long-lived PTS records. When released the MVCC GC queues go full throttle GC-ing as aggressively as possible, which we've seen to be disruptive to foreground traffic.

Describe the solution you'd like

• Issue MVCC GC requests with the low priority bit set for admission control to deprioritize when nodes are pushed to saturation (kvserver: lower priority level for mvcc gc work #85823);
• Introduce a cluster setting to pace the amount of MVCC GC queue processing (f00b130)
• Reproduce + roachtest-ify lack of performance isolation (latency) as a result of a large volume MVCC GC work. Hint: try creating a database with a large dataset (TPC-C with N warehouses), set a low default GC TTL, drop the DB, and let things rip. Failed attempts to create a small roachtest.
  • Evaluate CPU overhead and latency impact of subsequent merge queue activity (including generating/applying snapshots in preparation of the range merge, stats computation)
  • Evaluate overhead of secondary compactions when replicas are removed due to merge-queue related rebalancing
  • Note down other effects of post-MVCC activity
• Integrate MVCC GC work with elastic CPU tokens introduced in admission,kvserver: introduce an elastic cpu limiter #86638.
• (optional) Knob to disable MVCC GC queue cluster wide (poor man's pacing).
• (optional, very low priority, probably just a bad idea) Pacing knob (bytes/sec/store) if we have a large backlog of work pent up from super old protection records;

Aside: #84598 is tangentially related, motivated by the same incidents that motivated this issue. It considers how long garbage can be accumulated in the face of transient job failures. This issue covers the pacing needed independent of the amount of garbage that needs cleaning. That said, a short default retention window is good for many reasons, including the secondary effects of MVCC GC as mentioned above. Pacing all secondary effects is a larger endeavour (snapshots alone are covered in #80607) and out of scope for this issue.

Jira issue: CRDB-16754