Add Rocksdb as a db option. #3831
Description
We tried to use rocksdb as our storage db, see https://github.com/lino-network/tendermint/commits/yumin/rocksdb. The result shows that rocksDB, under our configuration, is better than cLevelDB in terms of total sync time and result db data size.
The rocksdb is configured with a large write buffer size and use zSTD as compression algorithm with dictionary compression enabled. We started a node with the above branch, and synced from height 0 to 3M+, (Lino network's data https://tracker.lino.network/#/).
Comparison with clevelDB
| | rocksdb | cleveldb |
|---------------|-----------|-----------|
| TotalDataSize | 64GB | 101GB |
| SyncTime | ~100Hours | ~170Hours |
We would like to add it as an option for db in tendermint. However, rocksDB is featured with very flexible configuration. Our configuration may not fit the needs of applications. So, there should be a way for application to configure it. Is the conf.toml file a good place for it? Or we just provide a general solution for all applications?
Our Rocksdb configuration:
[/column_family.cc:477] --------------- Options for column family [default]:
Options.comparator: leveldb.BytewiseComparator
Options.merge_operator: None
Options.compaction_filter: None
Options.compaction_filter_factory: None
Options.memtable_factory: SkipListFactory
Options.table_factory: BlockBasedTable
table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x3cdb370)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 1
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x3cce400
block_cache_name: LRUCache
block_cache_options:
capacity : 4294967296
num_shard_bits : 6
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 65536
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0
Options.write_buffer_size: 268435456
Options.max_write_buffer_number: 4
Options.compression: ZSTD
Options.bottommost_compression: ZSTD
Options.prefix_extractor: nullptr
Options.memtable_insert_with_hint_prefix_extractor: nullptr
Options.num_levels: 7
Options.min_write_buffer_number_to_merge: 1
Options.max_write_buffer_number_to_maintain: 0
Options.bottommost_compression_opts.window_bits: -14
Options.bottommost_compression_opts.level: 3
Options.bottommost_compression_opts.strategy: 0
Options.bottommost_compression_opts.max_dict_bytes: 262144
Options.bottommost_compression_opts.zstd_max_train_bytes: 10485760
Options.bottommost_compression_opts.enabled: true
Options.compression_opts.window_bits: -14
Options.compression_opts.level: 3
Options.compression_opts.strategy: 0
Options.compression_opts.max_dict_bytes: 262144
Options.compression_opts.zstd_max_train_bytes: 10485760
Options.compression_opts.enabled: true
Options.level0_file_num_compaction_trigger: 4
Options.level0_slowdown_writes_trigger: 12
Options.level0_stop_writes_trigger: 20
Options.target_file_size_base: 134217728
Options.target_file_size_multiplier: 1
Options.max_bytes_for_level_base: 1073741824
Options.level_compaction_dynamic_level_bytes: 1
Options.max_bytes_for_level_multiplier: 8.000000
Options.max_bytes_for_level_multiplier_addtl[0]: 1
Options.max_bytes_for_level_multiplier_addtl[1]: 1
Options.max_bytes_for_level_multiplier_addtl[2]: 1
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Options.max_bytes_for_level_multiplier_addtl[6]: 1
Options.max_sequential_skip_in_iterations: 8
Options.max_compaction_bytes: 3355443200
Options.arena_block_size: 33554432
Options.soft_pending_compaction_bytes_limit: 68719476736
Options.hard_pending_compaction_bytes_limit: 274877906944
Options.rate_limit_delay_max_milliseconds: 100
Options.disable_auto_compactions: 0
Options.compaction_style: kCompactionStyleLevel
Options.compaction_pri: kMinOverlappingRatio
Options.compaction_options_universal.size_ratio: 1
Options.compaction_options_universal.min_merge_width: 2
Options.compaction_options_universal.max_merge_width: 4294967295
Options.compaction_options_universal.max_size_amplification_percent: 200
Options.compaction_options_universal.compression_size_percent: -1
Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
Options.compaction_options_fifo.max_table_files_size: 1073741824
Options.compaction_options_fifo.allow_compaction: 0
Options.table_properties_collectors:
Options.inplace_update_support: 0
Options.inplace_update_num_locks: 10000
Options.memtable_prefix_bloom_size_ratio: 0.000000
Options.memtable_whole_key_filtering: 0
Options.memtable_huge_page_size: 0
Options.bloom_locality: 0
Options.max_successive_merges: 0
Options.optimize_filters_for_hits: 0
Options.paranoid_file_checks: 0
Options.force_consistency_checks: 0
Options.report_bg_io_stats: 0
Options.ttl: 0