We're experiencing apparently the same issue that we reported at #110
We having most of our bookies hang at 100% CPU while reading from rocksdb, but our disk metrics show little to no reads on the disks which is strange.

We're running pulsar 1.20 and bookkeeper version 4.3.1.72-yahoo.

Here is a jstack from a bookie:

jstack.txt

Our bookkeeper config is the following:

bk.conf.txt

This is currently affecting one of our production clusters and don't really know how to mitigate it.

@merlimat @sijie do any of you know how to mitigate or fix this, this is of course an issue with the rocksdb storage of bookkeeper.

I found this issue on rocksdb which talks about poor iterato4 performance when iterating through lots of tombstoned items, seems like this could be the case, or at least it expresses the same simptoms. We tried rebuilding the rocksdb index iusing bookkeeper cli and that seemed to work ok, maybe because that rebuilding made a clean index without tombstones?

Do you think this is the case? How could we mitigate this? In the issue they talk about caching the element just after the last deleted.