Issue

See #2748

pubsub currently has no buffer (capacity: 0) by default, which helps us identify issues early. One of such issues is if it (pubsub) has too many subscribers (for example, too many /broadcast_commit calls waiting for txs results), we can't accept a new msg until we've broadcasted a previous one (to all subscribers).

As an example, this

will block until previous msg is delivered to all subscribers.

This can potentially lead to consensus liveness issues.

Proposed action

We should rate limit the amount of broadcast_commit calls and other RPC endpoints which use pubsub (or recommend a limit for our clients to set in nginx or other software) + we need to figure out correct timeout for /broadcast_commit and make it configurable