The lock-table's wait-queues currently have no limit on the number of writers that can be queued on a given key. This can lead to unbounded queueing and diminishing quality of service for all writers as the queues build up. This is especially bad with high replication latency in multi-region clusters, as locks are held for the duration of a consensus replication round.

We should introduce a cluster setting that, when configured, limits the size that a single queue (lockWaitQueue.queuedWriters.Len()) can grow to before it begins rejecting new requests. If a request attempts to enter a queue that is already at this size limit, the request will be immediately rejected.

NOTE: Admission control and quality-of-service are deep topics and this issue doesn't attempt to provide a complete solution for them. Instead, it attempts to provide a limited, coarse-grained solution that we know will help serve as a release valve in various "hot key" scenarios.

A related item that this issue is not discussing is the queueing policy for these lock wait-queues. These queues are currently FIFO with a few caveats. This provides fairness, but also leads to quality of service degradation for all as queues grow. Furthermore, this leads to starvation when requests have a timeout that fires before any single request can get to the head of the queue. A LIFO queueing policy would help with this, though at the expense of fairness.

Alternatively, we could maintain fairness while avoiding starvation by introducing a dynamic maximum lock wait-queue depth based on request timeout duration. Given a request's maximum wait time before being canceled and the expected processing time of each item in the queue, we should be able to determine an optimal maximum wait-queue depth.

cc. @sumeerbhola. I'm going to try to prototype something here this Friday. Do we have an error type that we plan to use for request rejections?