Skip to content

OOM + no keys are stored #234

@github-dzwon-net

Description

@github-dzwon-net

Describe the bug

Error which I get:
OOM command not allowed when used memory > 'maxmemory'.

That happens after few days of using multimaster replication.

  • First keys are normally added and deleted after some time (ttl reached)
  • Server is using more and more RAM even if there is all the time a similar number of keys.
  • Server reach maxmemory and start to delete keys aggressively. That helps for some time, but number of stored keys is shrinking.
  • Server is using memory as much as it is allowed to (in this example 150MB - I lower that value to easier reproduce problem) and don't allow to set any new record and in this same time there are no other records at all!
  • I have situation when no records are stored, all allowed RAM is used, no new records are allowed.
  • In this situation helps only flushall or restart of cluster. Interesting is that flushdb didn't help.

Log Files
INFO

Server

redis_version:6.0.13
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:5055ac929b8ff5fb
redis_mode:standalone
os:Linux 4.15.0-108-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.3.0
process_id:14658
run_id:8ecca29d40f52576edc8b2b2f913a1c3cbba47a2
tcp_port:7305
uptime_in_seconds:251754
uptime_in_days:2
hz:10
configured_hz:10
lru_clock:6840703
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb.conf

Clients

connected_clients:3
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:4

Memory

used_memory:163565528
used_memory_human:155.99M
used_memory_rss:132898816
used_memory_rss_human:126.74M
used_memory_peak:164637640
used_memory_peak_human:157.01M
used_memory_peak_perc:99.35%
used_memory_overhead:56010024
used_memory_startup:54892488
used_memory_dataset:107555504
used_memory_dataset_perc:98.97%
allocator_allocated:164330880
allocator_active:165617664
allocator_resident:172400640
total_system_memory:16422928384
total_system_memory_human:15.30G
used_memory_lua:40960
used_memory_lua_human:40.00K
used_memory_scripts:216
used_memory_scripts_human:216B
number_of_cached_scripts:1
maxmemory:150000000
maxmemory_human:143.05M
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.01
allocator_frag_bytes:1286784
allocator_rss_ratio:1.04
allocator_rss_bytes:6782976
rss_overhead_ratio:0.77
rss_overhead_bytes:-39501824
mem_fragmentation_ratio:0.81
mem_fragmentation_bytes:-30665160
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:17186
mem_clients_normal:51558
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:1926415
rdb_bgsave_in_progress:0
rdb_last_save_time:1600424470
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:1392640
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:3472767
total_commands_processed:23512541
instantaneous_ops_per_sec:43
total_net_input_bytes:3279314670
total_net_output_bytes:814712820789
instantaneous_input_kbps:23.12
instantaneous_output_kbps:1.16
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:166307
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:359026
keyspace_hits:5331095
keyspace_misses:2955033
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:2104
migrate_cached_sockets:0
slave_expires_tracked_keys:1048577
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

Replication

role:active-replica
master_global_link_status:up
Master 0:
master_host:10.0.5.3
master_port:7305
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:17734421816
Master 1:
master_host:10.0.5.4
master_port:7305
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:25539788494
slave_priority:100
slave_read_only:0
connected_slaves:1
slave0:ip=10.0.5.3,port=7305,state=online,offset=4118827456,lag=1
master_replid:c78853e71f0bc62848f67fd9925eb45e59f25944
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:4118827785
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:4117779210
repl_backlog_histlen:1048576

CPU

used_cpu_sys:2325.346912
used_cpu_user:2016.994019
used_cpu_sys_children:0.023969
used_cpu_user_children:0.055929
server_threads:1
long_lock_waits:0

Modules

Cluster

cluster_enabled:0

Keyspace

MEMORY MALLOC-STATS
___ Begin jemalloc statistics ___
Version: "5.1.0-0-g0"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 16
opt.percpu_arena: "disabled"
opt.metadata_thp: "disabled"
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 10000 (arenas.muzzy_decay_ms: 10000)
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.lg_tcache_max: 15
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
Arenas: 16
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 164390288, active: 165724160, metadata: 6568056 (n_thp 0), resident: 172478464, mapped: 180645888, retained: 928747520
Background threads: 2, num_runs: 31928, run_interval: 13582024999 ns
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
background_thread 5507862 0 0 2 0 0 0
ctl 11015712 0 0 2 0 0 0
prof 0 0 0 0 0 0 0
Merged arenas stats:
assigned threads: 2
uptime: 251666694660044
dss allocation precedence: "N/A"
decaying: time npages sweeps madvises purged
dirty: N/A 59 31581 147541 7314137
muzzy: N/A 0 29512 105845 4868319
allocated nmalloc ndalloc nrequests
small: 80622992 8405522 6294495 235476554
large: 83767296 8146225 8146149 8146225
total: 164390288 16551747 14440644 243622779

active: 165724160
mapped: 180645888
retained: 928747520
base: 6502384
internal: 65672
metadata_thp: 0
tcache_bytes: 580216
resident: 172478464
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 5524116 0 0 3 0 0 0
extent_avail 12412743 1 5 52233 0 0 1
extents_dirty 28741225 5 73 63203 0 0 1
extents_muzzy 5927147 1 21 61327 0 0 1
extents_retained 5854500 14 42 56869 0 0 1
decay_dirty 5656895 0 4 64162 0 0 0
decay_muzzy 5646558 0 1 56218 0 0 0
base 11030380 0 0 5 0 0 0
tcache_list 5507860 0 0 3 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 3864 60159 59676 26538026 483 2 512 1 0.471 36580 23333 2 0 5567776 0 0 3 0 0 0
16 1 13408 68942 68104 22092829 838 4 256 1 0.818 40183 20538 4 0 5568575 0 0 3 0 0 0
24 2 25437600 3737599 2677699 56902848 1059900 2078 512 3 0.996 50923 41723 2511 232835 5604282 0 0 3 0 0 0
32 3 6176 36796 36603 33515362 193 11 128 1 0.137 17787 15742 70 978 5541618 0 0 3 0 0 0
40 4 5480 14171 14034 29602478 137 2 512 5 0.133 11543 7303 2 0 5526730 0 0 3 0 0 0
48 5 25166880 1242518 718208 19842599 524310 2052 256 3 0.998 33137 25747 3030 223790 5570749 0 0 3 0 0 0
56 6 29374464 1033922 509378 12792378 524544 1026 512 7 0.998 22001 21721 2018 951 5554595 0 0 3 0 0 0
64 7 5568 1584391 1584304 9379146 87 11 64 1 0.123 55855 39049 8413 30688 5619839 0 0 3 0 0 0
80 8 14080 4926 4750 4624112 176 4 256 5 0.171 2706 1314 4 501 5512015 0 0 3 0 0 0
96 9 6624 75741 75672 7464334 69 3 128 3 0.179 51283 19566 15 748 5578802 0 0 3 0 0 0
112 10 2352 6489 6468 29082 21 1 256 7 0.082 5916 5891 2827 0 5525328 0 0 3 0 0 0
128 11 3584 457687 457659 3373130 28 27 32 1 0.032 22403 23527 5028 147580 5563817 0 0 4 0 0 0
160 12 320 4166 4164 1405780 2 1 128 5 0.015 3122 3451 145 7 5514719 0 0 3 0 0 0
192 13 960 1676 1671 1116434 5 2 64 3 0.039 1480 1495 1274 0 5513378 0 0 3 0 0 0
224 14 2240 3845 3835 1581360 10 2 128 7 0.039 1568 2635 12 15 5512082 0 0 3 0 0 0
256 15 5120 346 326 325911 20 2 16 1 0.625 298 300 168 0 5508789 0 0 3 0 0 0
320 16 9600 6355 6325 2131163 30 5 64 5 0.093 3657 4331 48 620 5515936 0 0 3 0 0 0
384 17 1152 7017 7014 1917487 3 1 32 3 0.093 2971 4514 147 1409 5515635 0 0 3 0 0 0
448 18 6720 8211 8196 8170 15 1 64 7 0.234 3245 4732 53 2000 5515943 0 0 3 0 0 0
512 19 2048 8114 8110 532747 4 3 8 1 0.166 3339 4373 667 2640 5516911 0 0 3 0 0 0
640 20 7680 9879 9867 8802 12 1 32 5 0.375 3264 3693 237 992 5515330 0 0 3 0 0 0
768 21 11520 3077 3062 2890 15 3 16 3 0.312 1289 1479 162 155 5510964 0 0 3 0 0 0
896 22 25088 445 417 1032 28 1 32 7 0.875 266 289 7 7 5508464 0 0 3 0 0 0
1024 23 3072 803 800 1795 3 2 4 1 0.375 607 678 149 90 5509469 0 0 3 0 0 0
1280 24 5120 1512 1508 9401 4 1 16 5 0.250 908 968 94 152 5510016 0 0 3 0 0 0
1536 25 1536 5411 5410 60408 1 1 8 3 0.125 5063 5013 2218 127 5522439 0 0 3 0 0 0
1792 26 21504 5090 5078 54706 12 2 16 7 0.375 4823 4681 1629 271 5520659 0 0 3 0 0 0
2048 27 26624 4798 4785 52546 13 8 2 1 0.812 4591 4582 2467 95 5521975 0 0 3 0 0 0
2560 28 12800 723 718 3950 5 1 8 5 0.625 603 617 50 10 5509221 0 0 3 0 0 0
3072 29 6144 584 582 1731 2 1 4 3 0.500 493 517 49 14 5509002 0 0 3 0 0 0
3584 30 25088 350 343 498 7 2 8 7 0.437 248 260 26 12 5508436 0 0 3 0 0 0
4096 31 40960 4743 4733 51313 10 10 1 1 1 4674 4679 4743 0 5526713 0 0 3 0 0 0
5120 32 40960 108 100 219 8 4 4 5 0.500 54 67 29 6 5508050 0 0 3 0 0 0
6144 33 18432 80 77 115 3 2 2 3 0.750 37 49 44 7 5508041 0 0 3 0 0 0
7168 34 64512 91 82 205 9 3 4 7 0.750 45 56 21 9 5508015 0 0 3 0 0 0
8192 35 24576 4487 4484 50978 3 3 1 2 1 4444 4434 4487 0 5525723 0 0 3 0 0 0
10240 36 40960 98 94 357 4 2 2 5 1 53 64 50 9 5508089 0 0 3 0 0 0
12288 37 49152 93 89 134 4 4 1 3 1 67 51 93 0 5508172 0 0 3 0 0 0
14336 38 129024 79 70 98 9 5 2 7 0.900 48 52 44 7 5508052 0 0 3 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
16384 39 16384 145 144 204 1
20480 40 348160 36422 36405 3476906 17
24576 41 0 14 14 21 0
28672 42 0 267 267 458 0
32768 43 65536 53 51 96 2
40960 44 40960 5953515 5953514 5953515 1
49152 45 0 8123 8123 8123 0
57344 46 0 6895 6895 6895 0
65536 47 65536 10 9 10 1
81920 48 0 396 396 396 0
98304 49 0 23 23 23 0
114688 50 0 435 435 435 0
131072 51 0 16 16 16 0
163840 52 0 23 23 23 0
196608 53 0 75 75 75 0
229376 54 0 7 7 7 0
262144 55 0 9 9 9 0
327680 56 0 1 1 1 0
393216 57 0 1604628 1604628 1604628 0
458752 58 0 535086 535086 535086 0
524288 59 8388608 18 2 18 16
655360 60 655360 2 1 2 1
786432 61 12582912 29 13 29 16
917504 62 1835008 10 8 10 2
1048576 63 1048576 3 2 3 1
---
2097152 67 33554432 17 1 17 16
---
4194304 71 0 1 1 1 0
---
8388608 75 8388608 1 0 1 1
---
16777216 79 16777216 1 0 1 1
---
arenas[0]:
assigned threads: 1
uptime: 251666694660044
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 0 148 278 2165
muzzy: 10000 0 147 278 2165
allocated nmalloc ndalloc nrequests
small: 858104 18164 5504 26544
large: 54669312 179 125 179
total: 55527416 18343 5629 26723

active: 56266752
mapped: 68849664
retained: 9793536
base: 4653152
internal: 65672
metadata_thp: 0
tcache_bytes: 551992
resident: 60928000
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 2753997 0 0 2 0 0 0
extent_avail 2754614 0 0 240 0 0 0
extents_dirty 2755015 0 0 300 0 0 0
extents_muzzy 2754879 0 0 474 0 0 0
extents_retained 2755215 0 0 296 0 0 0
decay_dirty 2754879 0 0 657 0 0 0
decay_muzzy 2754779 0 0 655 0 0 0
base 5508301 0 0 4 0 0 0
tcache_list 2753930 0 0 2 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 2632 351 22 2893 329 1 512 1 0.642 4 1 1 0 2753938 0 0 2 0 0 0
16 1 10128 700 67 3060 633 3 256 1 0.824 8 2 3 0 2753934 0 0 2 0 0 0
24 2 265224 13200 2149 16140 11051 26 512 3 0.830 132 0 26 3 2754921 0 0 2 0 0 0
32 3 2240 300 230 313 70 2 128 1 0.273 7 2 2 1 2754042 0 0 2 0 0 0
40 4 2280 175 118 113 57 1 512 5 0.111 3 2 1 0 2753959 0 0 2 0 0 0
48 5 4080 125 40 70 85 1 256 3 0.332 2 2 1 0 2753933 0 0 2 0 0 0
56 6 4368 175 97 79 78 1 512 7 0.152 3 2 1 0 2753940 0 0 2 0 0 0
64 7 1984 736 705 1294 31 5 64 1 0.096 33 2 11 3 2754243 0 0 2 0 0 0
80 8 3520 500 456 750 44 2 256 5 0.085 13 2 2 1 2754079 0 0 2 0 0 0
96 9 3744 200 161 146 39 2 128 3 0.152 4 2 2 1 2754005 0 0 2 0 0 0
112 10 2352 125 104 18 21 1 256 7 0.082 2 2 1 0 2753944 0 0 2 0 0 0
128 11 128 33 32 6 1 1 32 1 0.031 1 3 2 0 2753936 0 0 3 0 0 0
160 12 320 100 98 4 2 1 128 5 0.015 1 3 1 0 2753933 0 0 2 0 0 0
192 13 768 64 60 2 4 1 64 3 0.062 1 2 1 0 2753932 0 0 2 0 0 0
224 14 2016 100 91 3 9 1 128 7 0.070 1 2 1 0 2753932 0 0 2 0 0 0
256 15 5120 32 12 20 20 2 16 1 0.625 2 1 2 0 2753933 0 0 2 0 0 0
320 16 6080 64 45 17 19 1 64 5 0.296 1 2 1 0 2753932 0 0 2 0 0 0
384 17 1152 32 29 6 3 1 32 3 0.093 1 2 1 0 2753932 0 0 2 0 0 0
448 18 6720 64 49 45 15 1 64 7 0.234 1 1 1 0 2753935 0 0 2 0 0 0
512 19 2048 25 21 23 4 3 8 1 0.166 3 1 4 3 2753948 0 0 2 0 0 0
640 20 7680 96 84 153 12 1 32 5 0.375 4 1 3 2 2753981 0 0 2 0 0 0
768 21 10752 40 26 172 14 2 16 3 0.437 3 1 3 1 2753954 0 0 2 0 0 0
896 22 25088 96 68 169 28 1 32 7 0.875 3 0 3 1 2753975 0 0 2 0 0 0
1024 23 3072 55 52 117 3 2 4 1 0.375 6 1 15 2 2753994 0 0 2 0 0 0
1280 24 5120 192 188 246 4 1 16 5 0.250 12 0 12 3 2754059 0 0 2 0 0 0
1536 25 0 120 120 178 0 0 8 3 1 12 0 15 5 2754041 0 0 2 0 0 0
1792 26 19712 64 53 68 11 1 16 7 0.687 4 0 4 1 2753981 0 0 2 0 0 0
2048 27 24576 45 33 48 12 7 2 1 0.857 5 1 23 3 2753992 0 0 2 0 0 0
2560 28 12800 70 65 73 5 1 8 5 0.625 7 0 9 4 2753997 0 0 2 0 0 0
3072 29 6144 60 58 71 2 1 4 3 0.500 6 0 15 5 2754001 0 0 2 0 0 0
3584 30 25088 30 23 28 7 2 8 7 0.437 3 0 4 1 2753958 0 0 2 0 0 0
4096 31 36864 40 31 47 9 9 1 1 1 4 0 40 0 2754030 0 0 2 0 0 0
5120 32 35840 30 23 29 7 3 4 5 0.583 3 0 8 2 2753962 0 0 2 0 0 0
6144 33 18432 15 12 11 3 2 2 3 0.750 2 1 8 2 2753957 0 0 2 0 0 0
7168 34 64512 30 21 33 9 3 4 7 0.750 3 0 8 2 2753962 0 0 2 0 0 0
8192 35 16384 20 18 11 2 2 1 2 1 2 0 20 0 2753985 0 0 2 0 0 0
10240 36 40960 20 16 61 4 2 2 5 1 2 0 10 5 2753965 0 0 2 0 0 0
12288 37 49152 20 16 17 4 4 1 3 1 3 1 20 0 2753983 0 0 2 0 0 0
14336 38 129024 20 11 10 9 5 2 7 0.900 2 0 10 2 2753957 0 0 2 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
16384 39 16384 7 6 7 1
20480 40 61440 20 17 20 3
24576 41 0 1 1 1 0
28672 42 0 54 54 54 0
32768 43 65536 11 9 11 2
40960 44 0 4 4 4 0
49152 45 0 5 5 5 0
57344 46 0 5 5 5 0
---
81920 48 0 10 10 10 0
---
114688 50 0 6 6 6 0
131072 51 0 1 1 1 0
163840 52 0 3 3 3 0
---
393216 57 0 3 3 3 0
458752 58 0 1 1 1 0
524288 59 8388608 16 0 16 16
---
786432 61 12582912 16 0 16 16
---
2097152 67 33554432 16 0 16 16
---
arenas[1]:
assigned threads: 1
uptime: 251666618656118
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 59 31433 147263 7311972
muzzy: 10000 0 29365 105567 4866154
allocated nmalloc ndalloc nrequests
small: 79764888 8387358 6288991 235450010
large: 29097984 8146046 8146024 8146046
total: 108862872 16533404 14435015 243596056

active: 109457408
mapped: 111796224
retained: 918953984
base: 1849232
internal: 0
metadata_thp: 0
tcache_bytes: 28224
resident: 111550464
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 2770119 0 0 1 0 0 0
extent_avail 9658129 1 5 51993 0 0 1
extents_dirty 25986210 5 73 62903 0 0 1
extents_muzzy 3172268 1 21 60853 0 0 1
extents_retained 3099285 14 42 56573 0 0 1
decay_dirty 2902016 0 4 63505 0 0 0
decay_muzzy 2891779 0 1 55563 0 0 0
base 5522079 0 0 1 0 0 0
tcache_list 2753930 0 0 1 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 1232 59808 59654 26535133 154 1 512 1 0.300 36576 23332 1 0 2813838 0 0 1 0 0 0
16 1 3280 68242 68037 22089769 205 1 256 1 0.800 40175 20536 1 0 2814641 0 0 1 0 0 0
24 2 25172376 3724399 2675550 56886708 1048849 2052 512 3 0.998 50791 41723 2485 232832 2849361 0 0 1 0 0 0
32 3 3936 36496 36373 33515049 123 9 128 1 0.106 17780 15740 68 977 2787576 0 0 1 0 0 0
40 4 3200 13996 13916 29602365 80 1 512 5 0.156 11540 7301 1 0 2772771 0 0 1 0 0 0
48 5 25162800 1242393 718168 19842529 524225 2051 256 3 0.998 33135 25745 3029 223790 2816816 0 0 1 0 0 0
56 6 29370096 1033747 509281 12792299 524466 1025 512 7 0.999 21998 21719 2017 951 2800655 0 0 1 0 0 0
64 7 3584 1583655 1583599 9377852 56 6 64 1 0.145 55822 39047 8402 30685 2865596 0 0 1 0 0 0
80 8 10560 4426 4294 4623362 132 2 256 5 0.257 2693 1312 2 500 2757936 0 0 1 0 0 0
96 9 2880 75541 75511 7464188 30 1 128 3 0.234 51279 19564 13 747 2824797 0 0 1 0 0 0
112 10 0 6364 6364 29064 0 0 256 7 1 5914 5889 2826 0 2771384 0 0 1 0 0 0
128 11 3456 457654 457627 3373124 27 26 32 1 0.032 22402 23524 5026 147580 2809881 0 0 1 0 0 0
160 12 0 4066 4066 1405776 0 0 128 5 1 3121 3448 144 7 2760786 0 0 1 0 0 0
192 13 192 1612 1611 1116432 1 1 64 3 0.015 1479 1493 1273 0 2759446 0 0 1 0 0 0
224 14 224 3745 3744 1581357 1 1 128 7 0.007 1567 2633 11 15 2758150 0 0 1 0 0 0
256 15 0 314 314 325891 0 0 16 1 1 296 299 166 0 2754856 0 0 1 0 0 0
320 16 3520 6291 6280 2131146 11 4 64 5 0.042 3656 4329 47 620 2762004 0 0 1 0 0 0
384 17 0 6985 6985 1917481 0 0 32 3 1 2970 4512 146 1409 2761703 0 0 1 0 0 0
448 18 0 8147 8147 8125 0 0 64 7 1 3244 4731 52 2000 2762008 0 0 1 0 0 0
512 19 0 8089 8089 532724 0 0 8 1 1 3336 4372 663 2637 2762963 0 0 1 0 0 0
640 20 0 9783 9783 8649 0 0 32 5 1 3260 3692 234 990 2761349 0 0 1 0 0 0
768 21 768 3037 3036 2718 1 1 16 3 0.062 1286 1478 159 154 2757010 0 0 1 0 0 0
896 22 0 349 349 863 0 0 32 7 1 263 289 4 6 2754489 0 0 1 0 0 0
1024 23 0 748 748 1678 0 0 4 1 1 601 677 134 88 2755475 0 0 1 0 0 0
1280 24 0 1320 1320 9155 0 0 16 5 1 896 968 82 149 2755957 0 0 1 0 0 0
1536 25 1536 5291 5290 60230 1 1 8 3 0.125 5051 5013 2203 122 2768398 0 0 1 0 0 0
1792 26 1792 5026 5025 54638 1 1 16 7 0.062 4819 4681 1625 270 2766678 0 0 1 0 0 0
2048 27 2048 4753 4752 52498 1 1 2 1 0.500 4586 4581 2444 92 2767983 0 0 1 0 0 0
2560 28 0 653 653 3877 0 0 8 5 1 596 617 41 6 2755224 0 0 1 0 0 0
3072 29 0 524 524 1660 0 0 4 3 1 487 517 34 9 2755001 0 0 1 0 0 0
3584 30 0 320 320 470 0 0 8 7 1 245 260 22 11 2754478 0 0 1 0 0 0
4096 31 4096 4703 4702 51266 1 1 1 1 1 4670 4679 4703 0 2772683 0 0 1 0 0 0
5120 32 5120 78 77 190 1 1 4 5 0.250 51 67 21 4 2754088 0 0 1 0 0 0
6144 33 0 65 65 104 0 0 2 3 1 35 48 36 5 2754084 0 0 1 0 0 0
7168 34 0 61 61 172 0 0 4 7 1 42 56 13 7 2754053 0 0 1 0 0 0
8192 35 8192 4467 4466 50967 1 1 1 2 1 4442 4434 4467 0 2771738 0 0 1 0 0 0
10240 36 0 78 78 296 0 0 2 5 1 51 64 40 4 2754124 0 0 1 0 0 0
12288 37 0 73 73 117 0 0 1 3 1 64 50 73 0 2754189 0 0 1 0 0 0
14336 38 0 59 59 88 0 0 2 7 1 46 52 34 5 2754095 0 0 1 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
16384 39 0 138 138 197 0
20480 40 286720 36402 36388 3476886 14
24576 41 0 13 13 20 0
28672 42 0 213 213 404 0
32768 43 0 42 42 85 0
40960 44 40960 5953511 5953510 5953511 1
49152 45 0 8118 8118 8118 0
57344 46 0 6890 6890 6890 0
65536 47 65536 10 9 10 1
81920 48 0 386 386 386 0
98304 49 0 23 23 23 0
114688 50 0 429 429 429 0
131072 51 0 15 15 15 0
163840 52 0 20 20 20 0
196608 53 0 75 75 75 0
229376 54 0 7 7 7 0
262144 55 0 9 9 9 0
327680 56 0 1 1 1 0
393216 57 0 1604625 1604625 1604625 0
458752 58 0 535085 535085 535085 0
524288 59 0 2 2 2 0
655360 60 655360 2 1 2 1
786432 61 0 13 13 13 0
917504 62 1835008 10 8 10 2
1048576 63 1048576 3 2 3 1
---
2097152 67 0 1 1 1 0
---
4194304 71 0 1 1 1 0
---
8388608 75 8388608 1 0 1 1
---
16777216 79 16777216 1 0 1 1
---
--- End jemalloc statistics ---

CONFIG GET*

  1. "rdbchecksum"
  2. "yes"
  3. "daemonize"
  4. "yes"
  5. "lua-replicate-commands"
  6. "yes"
  7. "always-show-logo"
  8. "no"
  9. "protected-mode"
  10. "yes"
  11. "rdbcompression"
  12. "yes"
  13. "rdb-del-sync-files"
  14. "no"
  15. "activerehashing"
  16. "yes"
  17. "stop-writes-on-bgsave-error"
  18. "yes"
  19. "dynamic-hz"
  20. "yes"
  21. "lazyfree-lazy-eviction"
  22. "no"
  23. "lazyfree-lazy-expire"
  24. "yes"
  25. "lazyfree-lazy-server-del"
  26. "no"
  27. "lazyfree-lazy-user-del"
  28. "no"
  29. "repl-disable-tcp-nodelay"
  30. "no"
  31. "repl-diskless-sync"
  32. "no"
  33. "aof-rewrite-incremental-fsync"
  34. "yes"
  35. "no-appendfsync-on-rewrite"
  36. "no"
  37. "cluster-require-full-coverage"
  38. "yes"
  39. "rdb-save-incremental-fsync"
  40. "yes"
  41. "aof-load-truncated"
  42. "yes"
  43. "aof-use-rdb-preamble"
  44. "yes"
  45. "cluster-replica-no-failover"
  46. "no"
  47. "cluster-slave-no-failover"
  48. "no"
  49. "replica-lazy-flush"
  50. "yes"
  51. "slave-lazy-flush"
  52. "yes"
  53. "replica-serve-stale-data"
  54. "yes"
  55. "slave-serve-stale-data"
  56. "yes"
  57. "replica-read-only"
  58. "no"
  59. "slave-read-only"
  60. "no"
  61. "replica-ignore-maxmemory"
  62. "yes"
  63. "slave-ignore-maxmemory"
  64. "yes"
  65. "multi-master"
  66. "yes"
  67. "jemalloc-bg-thread"
  68. "yes"
  69. "activedefrag"
  70. "yes"
  71. "syslog-enabled"
  72. "no"
  73. "cluster-enabled"
  74. "no"
  75. "appendonly"
  76. "no"
  77. "cluster-allow-reads-when-down"
  78. "no"
  79. "multi-master-no-forward"
  80. "no"
  81. "aclfile"
  82. ""
  83. "unixsocket"
  84. ""
  85. "pidfile"
  86. "/var/run/keydb/keydb-server.pid"
  87. "replica-announce-ip"
  88. ""
  89. "slave-announce-ip"
  90. ""
  91. "masteruser"
  92. ""
  93. "masterauth"
  94. ""
  95. "cluster-announce-ip"
  96. ""
  97. "syslog-ident"
  98. "redis"
  99. "dbfilename"
  100. "dump.rdb"
  101. "appendfilename"
  102. "appendonly.aof"
  103. "server_cpulist"
  104. ""
  105. "bio_cpulist"
  106. ""
  107. "aof_rewrite_cpulist"
  108. ""
  109. "bgsave_cpulist"
  110. ""
  111. "supervised"
  112. "no"
  113. "syslog-facility"
  114. "local0"
  115. "repl-diskless-load"
  116. "disabled"
  117. "loglevel"
  118. "notice"
  119. "maxmemory-policy"
  120. "allkeys-lru"
  121. "appendfsync"
  122. "everysec"
  123. "databases"
  124. "1"
  125. "port"
  126. "7305"
  127. "auto-aof-rewrite-percentage"
  128. "100"
  129. "cluster-replica-validity-factor"
  130. "10"
  131. "cluster-slave-validity-factor"
  132. "10"
  133. "list-max-ziplist-size"
  134. "-2"
  135. "tcp-keepalive"
  136. "300"
  137. "cluster-migration-barrier"
  138. "1"
  139. "active-defrag-cycle-min"
  140. "1"
  141. "active-defrag-cycle-max"
  142. "25"
  143. "active-defrag-threshold-lower"
  144. "10"
  145. "active-defrag-threshold-upper"
  146. "100"
  147. "lfu-log-factor"
  148. "10"
  149. "lfu-decay-time"
  150. "1"
  151. "replica-priority"
  152. "100"
  153. "slave-priority"
  154. "100"
  155. "repl-diskless-sync-delay"
  156. "15"
  157. "maxmemory-samples"
  158. "5"
  159. "timeout"
  160. "0"
  161. "replica-announce-port"
  162. "0"
  163. "slave-announce-port"
  164. "0"
  165. "tcp-backlog"
  166. "4096"
  167. "cluster-announce-bus-port"
  168. "0"
  169. "cluster-announce-port"
  170. "0"
  171. "repl-timeout"
  172. "60"
  173. "repl-ping-replica-period"
  174. "1"
  175. "repl-ping-slave-period"
  176. "1"
  177. "list-compress-depth"
  178. "0"
  179. "rdb-key-save-delay"
  180. "0"
  181. "key-load-delay"
  182. "0"
  183. "active-expire-effort"
  184. "1"
  185. "hz"
  186. "10"
  187. "min-replicas-to-write"
  188. "0"
  189. "min-slaves-to-write"
  190. "0"
  191. "min-replicas-max-lag"
  192. "10"
  193. "min-slaves-max-lag"
  194. "10"
  195. "min-clients-per-thread"
  196. "50"
  197. "maxclients"
  198. "60000"
  199. "active-defrag-max-scan-fields"
  200. "1000"
  201. "slowlog-max-len"
  202. "128"
  203. "acllog-max-len"
  204. "128"
  205. "lua-time-limit"
  206. "5000"
  207. "cluster-node-timeout"
  208. "15000"
  209. "slowlog-log-slower-than"
  210. "10000"
  211. "latency-monitor-threshold"
  212. "0"
  213. "proto-max-bulk-len"
  214. "536870912"
  215. "stream-node-max-entries"
  216. "100"
  217. "repl-backlog-size"
  218. "1048576"
  219. "maxmemory"
  220. "150000000"
  221. "hash-max-ziplist-entries"
  222. "2048"
  223. "set-max-intset-entries"
  224. "512"
  225. "zset-max-ziplist-entries"
  226. "256"
  227. "active-defrag-ignore-bytes"
  228. "104857600"
  229. "hash-max-ziplist-value"
  230. "256"
  231. "stream-node-max-bytes"
  232. "4096"
  233. "zset-max-ziplist-value"
  234. "128"
  235. "hll-sparse-max-bytes"
  236. "3000"
  237. "tracking-table-max-keys"
  238. "1000000"
  239. "repl-backlog-ttl"
  240. "3600"
  241. "auto-aof-rewrite-min-size"
  242. "67108864"
  243. "tls-port"
  244. "0"
  245. "tls-cluster"
  246. "no"
  247. "tls-replication"
  248. "no"
  249. "tls-auth-clients"
  250. "yes"
  251. "tls-prefer-server-ciphers"
  252. "no"
  253. "tls-cert-file"
  254. ""
  255. "tls-key-file"
  256. ""
  257. "tls-dh-params-file"
  258. ""
  259. "tls-ca-cert-file"
  260. ""
  261. "tls-ca-cert-dir"
  262. ""
  263. "tls-protocols"
  264. ""
  265. "tls-ciphers"
  266. ""
  267. "tls-ciphersuites"
  268. ""
  269. "logfile"
  270. "/var/log/keydb/keydb-server.log"
  271. "client-query-buffer-limit"
  272. "1073741824"
  273. "watchdog-period"
  274. "0"
  275. "dir"
  276. "/var/lib/keydb"
  277. "save"
  278. ""
  279. "client-output-buffer-limit"
  280. "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
  281. "unixsocketperm"
  282. "0"
  283. "slaveof"
  284. "10.0.5.3 7305"
  285. "slaveof"
  286. "10.0.5.4 7305"
  287. "notify-keyspace-events"
  288. ""
  289. "bind"
  290. "10.0.5.2 172.17.0.1"
  291. "requirepass"
  292. ""

To Reproduce
It's hard to reproduce. It require hours of feeding DB with data. In my case there are both short records used as sessions and long records like SQL query cache.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions