Feature Request

Summary

Whenever a peer is lagging by more than 1 block, we load the commit for the height they were on and send them the whole commit (per code comment. Admittedly I haven't looked long, but I don't see how it's currently sending the whole commit vs just a single vote from it). Source

Loading the commit requires doing a db read and a proto unmarshal. Source In benchmarks we see the proto unmarshal taking a large number of heap allocations, and this is likely duplicate work. (Unmarshalling the same block commit)

We should keep a small LRU cache for height to unmarshalled commit to reduce the memory pressure here. I suspect the same maybe helpful for block parts as well.

In v37 we saw that this was around 70GB of heap allocation on a query serving node over 1-2 hours. (Will paste memprofile later)

I do not know the total system impact it has, but a 70GB ram allocation is very high.

Proposal

Add a simple lru cache in blockstore for loading commit and block parts