Storage optimization #9881
Description
High-level tracking issue for general storage optimization efforts. This issue can be expanded over time.
At present (late 2022), depending on their configuration, Tendermint-based nodes use large quantities of storage space. This has significant cost implications for operators. We aim to implement strategies to reduce and/or offload certain data stored in order to reduce operators' costs.
- Investigate which types of data are consuming the most storage space
- Provide a way to obtain a clear breakdown of the different types of data a node is storing and how much storage they use.
- Define storage use cases and workloads #9943 Preliminary investigation to identify the users and workloads of the storage backend. The result of this step is supposed to be a high level list containing information on who and how (at a very high level) accesses the storage.
- What components of Tendermint access the storage and how do they access it (read/write)
- How are our users interacting with the storage (operators, relayers etc.)
- Extend the previous point with: Establish and implement the relevant metrics to understand storage workloads #9773
- Define the metrics we need to understand the usage of storage.
- Implement the metrics defined.
- Simplify Tendermint database engine support #9749
- Gather data using previously implemented metrics to identify the storage requirements for Tendermint.
- Research different databases based on the obtained requirements.
- Refactor Tendermint to use a single database backend.
- Identify the functionalities we want to support within Tendermint and offload non-critical data and functionality
- ADR-082: Data Companion Push API #9437
- Provide a mechanism to offload event indexing based on ADR-082: Data Companion Push API #9437 or a different design.
- Short-term fixes to alleviate pressure for operators
- Develop an understanding on how to manage a blockchain's history across the whole network
- Storage management is complex (a UX/docs-related set of problems)