Feature Request

Summary

Rechecking all txs in the mempool seems to block consensus. We see this from:

• How pprof's look
• How the code appears

This is problematic as it means larger mempools will delay consensus longer. (Also IBC has a change that creeped in that is causing overly large expenses in RecheckTx)

Here is a copy of a pprof from a live-syncing Osmosis full node during 1 hour, with relatively average tx volume:

We see that it blocks Commit right here: https://github.com/cometbft/cometbft/blob/main/state/execution.go#L419-L426

If you look into the relevant code, each recheck call is actually synchronous due to how the callback's are structured.

Problem Definition

We should make the mempool rechecking not block BlockExecutor.ApplyBlock.

Ideally it should only be blocking ProposeBlock until either everything in the mempool is rechecked or blockGas worth of txs are rechecked. It should never be blocking for timeout_prevote