To save state and block data, CometBFT generates keys with string + height prefixes. They are at the moment internally sorted lexicographically.

This potentiall impacts both data access time (lookup) as well as the performance of compaction.

Some work in Q2 2023 indicates that changing the way keys are represented and forcing them to be ordered by height improved compaction and the disk footprint of the blockstore (28MB shrank to 18MB). This was however a POC and the generated data size was probably not big enough to be certain of gains.

Also, it is not 100% clear whether sorting all types of data by height is desired. Talking to users and analyzing the access patterns should help with that.

We need to:

• Understand which data is access together to make sure the keys are in a layout that is beneficial for common access patterns. For example, we often fetch the last commits of a previous height and those should be located nearby.
• feat(store)!: Support for different key layout #2327 [ ] Reconsider representation of keys for state and block store #1041 (Adding support for a different key ordering. Previous work done on this topic for Tendermint 0.35 and 0.36:
  store: order-preserving varint key encoding tendermint/tendermint#5771 . It is ported into this work. )

This would potentially resolve or improve informalsystems/interchain#1

DoD

• Understand the way data is layout by the database
• Decide on optimal key representation to suit our use cases
• Benchmark the code with a different layout against the baseline in Storage evaluation baseline: measure and report current storage related behaviour of CometBFT #67
• Evaluate the impact on pruning and compaction
• Update documentation
• Write tests to confirm behaviour is correct
• Provide users with a tool to migrate to the new data layout in case one is chosen.
• Remove mtx, base, height fields from the BlockStore struct https://github.com/cometbft/cometbft/pull/1856/files#r1430831931