As we know tendermint is fast blocking consensus protocol, the height of blockchain grows quickly.

Some users would like to replay the blocks from genesis, and the speed is around 20 blocks/ per second, which is slow, and the bottleneck is storage according to our analysis.

Here is our solution:

1. Let users download all the data(including history blocks).
2. Delete application.db
3. The application db use mem db, and the rest use leveldb, then start the tendermint client.

However, the performance does not improve, and the speed is still around 20 blocks per second. We profile the CPU usage:

Each time commit a block wil save SaveOrphans:

func (ndb *nodeDB) SaveOrphans(version int64, orphans map[string]int64) {
	....
	toVersion := ndb.getPreviousVersion(version)
        ....

func (ndb *nodeDB) getPreviousVersion(version int64) int64 {
	itr := ndb.db.ReverseIterator(
		rootKeyFormat.Key(1),
		rootKeyFormat.Key(version),
	)
	...

	return 0
}

func (db *MemDB) ReverseIterator(start, end []byte) Iterator {
         ....
	keys := db.getSortedKeys(start, end, true)
	return newMemDBIterator(db, keys, start, end)
}

func (db *MemDB) getSortedKeys(start, end []byte, reverse bool) []string {
	keys := []string{}
	for key := range db.db {
                 inDomain := IsKeyInDomain([]byte(key), start, end)
		....
	}
	sort.Strings(keys)
         ...
        reurn keys
}

And let's see what is memdb's structure:

type MemDB struct {
	mtx sync.Mutex
	db  map[string][]byte
}

It means it will iterate all the keys in db when doing SaveOrphans, obviously IsKeyInDomain and sort.Strings(keys) functions are CPU intensive.

So do you guys have any plan to improve the performance of Memdb? Or any idea to solve the problem that needs a long time to replay from genesis? (There is state sync mode can skip history blocks and sync the state snapshot, but that is not what we want. )