Feature Request

Summary

Same as #3154 , but for block parts. Here saving the WAL writes is important.

Osmosis blocks are most of the time 1-2 block parts, but we sometimes get spikes to much higher.

Here is a graph of late block parts (green), and num_block_parts * 1000 (yellow). The *1000, was to make these visible on the same axes

Ignore the resets to 0 for the green, thats us resetting the node entirely.

So the late block parts seems to have odd trends, e.g. for 7 days we were at 17k parts per day, in a range hwere we only had 1-2 block parts. Sometimes its better, sometimes its worse. During the spike to 120 parts in ~15 minutes (6 blocks with 10-20 parts, 1 block with 30 parts, and several with 5 parts), we had many (~5k) late parts received!

Precisely when were getting all these late parts, is likely when its most important for us to save WAL load of writing these to disk, and times under mutex.

Problem Definition

Progress to lowering our contention on the consensus mutex when under load.

Proposal

Essentially same implementation as #3157 but for block parts