The result of this issue should help to answer the question: "how do new releases, e.g. v0.37, compare to v0.34 in terms of performance and stability?"

Run a set of two testnets and collect the following on the v0.34.x kv store application for each:

1. All Prometheus metrics for the full duration of each execution so that we can do post-hoc analysis on the metrics. The kinds of metrics in which we're interested include, but are not limited to, the following:
   1. Consensus rounds per height
   2. Maximum connected peers, Minimum connected peers, Rate of change of peer connections
   3. Memory resident set size
   4. CPU utilization
   5. Blocks produced per minute
   6. Seconds for each step of consensus (Propose, Prevote, Precommit, Commit)
2. Transaction latency while varying the throughput of a client submitting transactions into a single node, with that client running on the same machine as that node. Throughput must be gradually increased until we reach saturation. In order to do this, we need the following:
   • qa: Transaction submission tool for latency estimation for kvstore app #9330
   • qa: Tool to estimate transaction latency from block data for kvstore app #9331
   • Infrastructure/configuration to run the load generation tool, once the network has stabilized, with different transaction throughput rates.
3. Stretch goal: capture the network topology for each testnet run. In order to do this, we need:
   • qa: Topology scanning for testnet execution #9332

200 Node Testnet

Run a 200 node testnet with 5 seed nodes, 175 validators, and 20 non-validating full nodes. Run this for an hour and collect metrics. Next, restart the network but apply load to the network and collect the above metrics.

Rotating Node Testnet

Real-world deployments of Tendermint frequently see new nodes arrive and old nodes exit the network. To test this, we will run a network with 10 validators and 3 seed nodes. A rolling set of 25 full nodes are started and each connects to the network by dialing one of the seed nodes. Once the node is able to sync to the head of the chain and begin producing blocks, it is stopped. Once stopped a new node takes its place.

Dependencies:

• Port ABCI Client metrics to main #9165
• Port consensus timing metrics to main branch #9166