We have encountered scenarios with a large number of goroutines, which often causes an increase in the runnable goroutines, while the mean CPU utilization stays low (sometimes as low as 25%). Since there are non-zero runnable goroutines, at very short time scales of a few ms CPU utilization must be 100%. Since admission control (AC) samples the runnable goroutine count every 1ms, in order to react to such short time scales, we do see some drop in the slot count in some of these cases, and at the same time queueing in the AC queues. The concern that comes up when seeing such queueing is whether AC is making the situation worse in its attempt to shift some queueing from the goroutine scheduler into the AC queue. Note that since admission.kv_slot_adjuster.overload_threshold is set to 32, AC does allow for significant queuing in the goroutine scheduler too, in an attempt to be work conserving.

We try to answer two questions:
Q1. Should such scenarios be considered unreasonable and be fixed outside AC. There are 2 cases we have seen:

• Internal work that is CPU bound, specifically rangefeed goroutines: This is an unreasonable situation since regardless of AC behavior, we are piling up goroutines in the goroutine scheduler which will increase scheduling latency for user facing work. This is discussed in kvserver,cdc: high rangefeed count leads to scheduler overload #96395.
• High concurrency of work submitted by the user: This came up in https://github.com/cockroachlabs/support/issues/2045#issuecomment-1412479981, https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1675366778810919. Given that we have a distributed SQL database which can distribute work across many nodes and work can block due to contention, it is reasonable for a user to increase concurrency in order to not leave any resources unused. That is, it is reasonable for the full available user concurrency to be exposed to CockroachDB, and for CockroachDB to reduce that concurrency only if it is approaching some practical limit in number of Goroutines, SQL connections, outstanding RPCs etc.

Q2. Given that these scenarios are sometimes reasonable, can we add metrics to answer the concern mentioned earlier regarding whether AC is making the situation worse.

The slot mechanism is imposing a max concurrency. If the max concurrency leaves some CPU idle, because enough of the admitted work is blocked (contention or IO), while we have work queued in AC, the AC queueing is not work conserving. We can try to sample this at 1ms intervals the way we sample numRunnableGoroutines.
If AC is indeed work conserving, AC queueing while the CPU "seems underutilized" is not happening, since the CPU is fully utilized when there is queueing in AC.

Jira issue: CRDB-24153