Compensate for memory peak at flush time #10126
Conversation
|
Please tag as 0.14.1. |
|
@sipa are any of your experiments relatively simple to reproduce? I'd like to step through the high usage spots to get a better feel for what's happening. |
|
@theuni Using the top commit at https://github.com/sipa/bitcoin/commits/repmemusage I've done tests with reindexes and syncs from scratch, killing and restarting bitcoind at certain memory usage points. Then looked at (MAX_RSS - 280MB) / (last_reported_cache=_line). |
|
utACK 7228ce8 |
7228ce8 Compensate for memory peak at flush time (Pieter Wuille) Tree-SHA512: 97e9848410fab061402c85d8440c54a50dd8a0203b2ea194013ea116700a6dc1b4b26b8c5f9c9c68c1f5c6b935c5d6c737437c1911b003d9ff5445c570cd449d
|
@sipa I still object to this PR without at least some increase in the default dbcache. I believe that in the event the mempool is full and there is no loan available then performance will be unnecessarily poor. I think 300 MB (with this new calculation) is approximately the bare minimum we can achieve without likely exceeding it by accident when connecting a block. It is not appropriate for a default. |
7228ce8 Compensate for memory peak at flush time (Pieter Wuille) Tree-SHA512: 97e9848410fab061402c85d8440c54a50dd8a0203b2ea194013ea116700a6dc1b4b26b8c5f9c9c68c1f5c6b935c5d6c737437c1911b003d9ff5445c570cd449d
It seems that our conversion from CCoinsCacheDB into a CDBBatch, and the following processing of the batch by LevelDB causes a spike in memory usage. In various experiments (all on x86_64, though), I've determined that this spike is between x1.5 and x1.9.
Take this into account ahead of time before deciding whether to flush or not.