Skip to content

Clarification on max_total_wal_size #5789

@adamretter

Description

@adamretter

I understand that max_total_wal_size is defined per-database. The C++ doc describes that when set to 0 it should be calculated dynamically as:

If set to 0
// (default), we will dynamically choose the WAL size limit to be
// [sum of all write_buffer_size * max_write_buffer_number] * 4

Using RocksDB git HEAD at 2019-06-23, if I set max_total_wal_size = 0, and then for each of my 15 CFs (Column Families) set:

write_buffer_size = 128 MB
max_write_buffer_number = 6

We don’t do any manual flushes, and instead we are relying on the max_total_wal_size formula to flush the Column families when the WAL exceeds its setting.

After loading about 10GB of data across about the column families into a clean database, and then leaving the system to idle for some hours, I am still seeing *.log files on disk totalling about ~20GB.

Unfortunately, I believe either one of two things is happening:

  1. I am misunderstanding how the above dynamic calculation works, or
  2. there is a bug whereby max_total_wal_size is not dynamically calculated correctly.

I was expecting the flush and WAL cleanup to happen at: (128MB * 6) * 4 = 3GB.

I am wondering how to interpret the text "sum of all write_buffer_size". Should it actually mean the total WAL size across all column families instead, i.e.: (15 CFs * (128MB * 6)) * 4 = 45GB ?

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions