-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Description
I understand that max_total_wal_size is defined per-database. The C++ doc describes that when set to 0 it should be calculated dynamically as:
If set to 0
// (default), we will dynamically choose the WAL size limit to be
// [sum of all write_buffer_size * max_write_buffer_number] * 4
Using RocksDB git HEAD at 2019-06-23, if I set max_total_wal_size = 0, and then for each of my 15 CFs (Column Families) set:
write_buffer_size = 128 MB
max_write_buffer_number = 6
We don’t do any manual flushes, and instead we are relying on the max_total_wal_size formula to flush the Column families when the WAL exceeds its setting.
After loading about 10GB of data across about the column families into a clean database, and then leaving the system to idle for some hours, I am still seeing *.log files on disk totalling about ~20GB.
Unfortunately, I believe either one of two things is happening:
- I am misunderstanding how the above dynamic calculation works, or
- there is a bug whereby
max_total_wal_sizeis not dynamically calculated correctly.
I was expecting the flush and WAL cleanup to happen at: (128MB * 6) * 4 = 3GB.
I am wondering how to interpret the text "sum of all write_buffer_size". Should it actually mean the total WAL size across all column families instead, i.e.: (15 CFs * (128MB * 6)) * 4 = 45GB ?