Describe the problem

We have received numerous instances in which large numbers of intents (hundreds of thousands and more) were able to build up on a range. While this is always theoretically possible - intent cleanup is async and best-effort, so an ill-timed crash can temporarily leave the intents behind - it appears that this is happening in scenarios where it is unexpected.

See for example (internal link): https://github.com/cockroachlabs/support/issues/817

The issue often leads to stuck backups because of #59704 (which we should independently fix).

To Reproduce

Reproducing these issues is part of the problem. So far we only strongly suspect that there are issues, but there is no concrete clue.

Expected behavior
Intents should only be able to build up when

Additional data / screenshots
Recap of the intent resolution behavior:

Intents are generally resolved when EndTxn(commit=true) has finished applying. We will then hit this code path:

which calls into this:

One possibility is that the intent resolver erroneously skips large batches of txn intents because it is already tracking some smaller amount of work on behalf of the same txn, in which case we'd somehow end up in the !locked code path. In practice, this seems very unlikely, since we only invoke this method from the GC queue (which doesn't touch txns younger than 1h) and the EndTxn(commit=true) proposal result itself, of which there is only ever one.

This raises the question why lockInflightTxnCleanup even exists. I think it may either be premature optimization or related to the GC queue code path. The code hasn't changed much since 2017, i.e. it is very old.

I did check that the cleanup intents are properly scoped to a new background-derived context, i.e. we're not accidentally tying these cleanup attempts to the short-lived caller's context. We're also not introducing an overly aggressive timeout, actually to the best of my knowledge in this particular path there isn't a timeout (?) as runAsyncTask derives from context.Background() and we don't add a timeout anywhere.

Also, isn't this code wrong:

It seems that it would erase ba.sendDeadline if len(ba.Reqs) > 0, ba.sendDeadline > 0, and !rHasDeadline. However, if anything, this would swallow a context cancellation rather than cause a premature one.

Another thing worth noting is that parallel commits prevents clients from seeing the backpressure. When EndTxn(commit=true) runs, it may feel backpressure. However, that request in parallel commits is completely disjoint from the SQL client as it is fired-and-forgotten here:

Environment:

• CockroachDB 20.1.8 and above (probably also below)

Additional context
Once we have the separated lock table, it might be much cheaper to be reactive to the problem.

Epic: CRDB-2554