The genesis file is stored as a value in CometBFTs kv store. As the file can be large it can hit the internal file size limitations of databases.

We need to figure out an alternative representation of this file. A proposal was to store the hash of the genesis file in the state store but the actual file is stored outside the store.

Existing PRs targetting this :

• Remove genesis persistence in state db #1017
  Historically, this issue was mentioned in the original Tendermint repository under:
  • genesis file may go over the recommended maximum value size tendermint/tendermint#8416
  • node: fix genesis on start up tendermint/tendermint#6563

A potentially related problem is that at the moment the Genesis file is retrievable via RPC. Even though it is retrieved in chunks, it is still loaded entirely into memory before it is chunked up. We could improve on this if we take it into consideration.

Things to understand and complete as part of this issue:

• Determine "How" and "How Often" is the genesis file accessed? #1286
• What is the typical size of the genesis file? #1078

After discussing, this issue will be addressed in two phases:

Phase I

• Remove the genesis file from the kvstore and store it in an external file.
  • Adopt PR Remove genesis persistence in state db #1017.
  • Change the hash on the doc, to a hash on the genesis file itself. #1287
  • Ensure smooth upgrades after removing genesis from kvstore. #1259

Phase II

• Reduce genesis's memory footprint: Do not keep the entire genesis file in memory during node operation #1290

Phase III

• Optimize chunking logic (from storage and RPC perspective): Optimize genesis file chunking #1289
  • Reduce genesis chunks size: Chore(RPC): Reduce Genesis Chunks Size #4298

DoD

• Genesis file is not stored as one entry in the state store
• Tests written to confirm no errors are generated for large genesis files
• Document how the genesis file is stored and represented
• (stretch) Improve retrieval of Genesis file when requested via RPC