The pubsub system is used by both the consensus and state packages to publish events.

Events in the state package only happen when we've committed a new block, and include all the information from that block. These are the main events necessary for users, and are what the block explorers are made from. It's also what we persist in the blockstore, state, and tx indexing dbs.

Events in the consensus package are real-time consensus events: receiving complete proposal and block, receiving a vote, locking/unlocking, timing out, etc. They are also used for the consensus state machine tests. While they are currently exposed for subscription over websocket, they arguably should not be, or at least not in the same way that the state package events are. We do not persist these anywhere, but they are re-computable from the information saved in the consensus WAL.

While we have plans to refactor the consensus state to avoid needing a pubsub to properly unit test it (ADR-030), in the meantime we should distinguish between these types of events as follows:

• state events happen asynchronously and do not block the consensus
  • slow clients will be dropped and can fetch missed information from the rpc - see Bucky/pubsub #3209
• consensus events happen synchronously and do block the consensus
• restrict subscribtions to consensus events.

This way we can try to support lots of subscriptions to user-level events, where we don't block the consensus on them, and where if they can't keep up with kill the subscription and let them recover by getting missed information from the rpc. It also lets us continue to use the consensus events to block the consensus, without worrying that users can abuse it, because we restrict which users can even subscribe to these in the first place.

The first two points (async vs sync events) will depend on ADR-033, especially the different treatment of buffered and unbuffered channels proposed in #3209.

The third point, of restricting subscriptions to consensus events, has a few options:

• config option to make them available over generic websocket
• config option to make them available over generic websocket to a specific ClientID only (eg. should be a 20-byte random string)
• config option to make them only available on a separate RPC port

Thoughts on the best approach for restricting access to these consensus events?

We can also eventually consider removing the need to publish the state events as independent events directly from state package to subscribers; instead, we can notify some service in the RPC package that a new block has been committed, and let it handle publishing to all the clients. The notification could be a single event with all the information, or it could just contain a height, in which case the new service could fetch all the relevant information from the various stores. This could improve the separation of concerns between packages, and might also provide a more generic "new block notification system" internally that can address some of the weirdness of the Update methods on the mempool and evidence pool.