Skip to content

[BUG] used_memory exceeds maxmemory #601

@CrazyTennisFan

Description

@CrazyTennisFan

Recently, when I test KeyDb key eviction, I found if maxmemory and repl-backlog-size are included in the profile,
finally, the used_memory will be the sum of maxmemory and repl-backlog-size and used_memory will be far more than maxmemory .
However, if I do the same test in Redis 6.2.6, the used_memory finally will be equal to the maxmemory.

Note: in my test, there is no AOF, RDB, and SSD file.

Test command:

./memtier_benchmark -s 127.0.01 -p 6381 -t 10 -c 100 -n 10000000000 -d 506 --key-minimum=1 --key-maximum=10000000000 --ratio 1:0 --key-pattern=P:P

Test config file:

bind 127.0.01
port 6381
protected-mode no
notify-keyspace-events Ex
tcp-keepalive 30
timeout 0
maxmemory 8589934592
maxclients 10010
save ""
unixsocket /tmp/keydb.sock
unixsocketperm 600
client-output-buffer-limit slave 2147483648 2147483648 60
tcp-backlog 10000
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes
slave-lazy-flush yes
rdbcompression no
databases 16
appendonly no
appendfilename appendonly.aof
appendfsync no
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 0
lazyfree-lazy-user-flush yes
maxmemory-policy allkeys-lru
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
latency-monitor-threshold 0
repl-backlog-size 3221225472
repl-backlog-ttl 3600
slowlog-log-slower-than 10000
slowlog-max-len 128

Test results in Keydb:

Memory

used_memory:11811159360
used_memory_human:11.00G
used_memory_rss:12164382720

used_memory_rss_human:11.33G
used_memory_peak:11812731152
used_memory_peak_human:11.00G
used_memory_peak_perc:99.99%
used_memory_overhead:1080273498
used_memory_startup:5186784
used_memory_dataset:10730885862
used_memory_dataset_perc:90.89%
allocator_allocated:11809899824
allocator_active:11828928512
allocator_resident:12212736000
total_system_memory:16818872320
total_system_memory_human:15.66G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:8589934592
maxmemory_human:8.00G
maxmemory_policy:allkeys-lru

allocator_frag_ratio:1.00
allocator_frag_bytes:19028688
allocator_rss_ratio:1.03
allocator_rss_bytes:383807488
rss_overhead_ratio:1.00
rss_overhead_bytes:-48353280
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:355311544
mem_not_counted_for_evict:3221225472
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20655770
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none

Test in Redis 6.2.6

Memory

used_memory:8589934432
used_memory_human:8.00G
used_memory_rss:8817569792
used_memory_rss_human:8.21G

used_memory_peak:8589934664
used_memory_peak_human:8.00G
used_memory_peak_perc:100.00%
used_memory_overhead:738410064
used_memory_startup:810072
used_memory_dataset:7851524368
used_memory_dataset_perc:91.41%
allocator_allocated:8589978384
allocator_active:8603328512
allocator_resident:8875233280
total_system_memory:16818872320
total_system_memory_human:15.66G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:8589934592
maxmemory_human:8.00G
maxmemory_policy:allkeys-lru

allocator_frag_ratio:1.00
allocator_frag_bytes:13350128
allocator_rss_ratio:1.03
allocator_rss_bytes:271904768
rss_overhead_ratio:0.99
rss_overhead_bytes:-57663488
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:227635720
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20532504
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

I found there is another issue when I did the test:
If I enable RDB or FLUSH, the used_memory will be greater than the maxmemory.
Please check the following situation:

Enable RDB only, maxmeory 8GB, repl-backlog-size 1GB

Memory

used_memory:10849729320
used_memory_human:10.10G
used_memory_rss:11168501760
used_memory_rss_human:10.40G

used_memory_peak:10850031880
used_memory_peak_human:10.10G
used_memory_peak_perc:100.00%
used_memory_overhead:770008330
used_memory_startup:5191488
used_memory_dataset:10079720990
used_memory_dataset_perc:92.95%
allocator_allocated:10850758976
allocator_active:10860331008
allocator_resident:11215519744
total_system_memory:16777809920
total_system_memory_human:15.63G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:8589934592
maxmemory_human:8.00G
maxmemory_policy:allkeys-lru

allocator_frag_ratio:1.00
allocator_frag_bytes:9572032
allocator_rss_ratio:1.03
allocator_rss_bytes:355188736
rss_overhead_ratio:1.00
rss_overhead_bytes:-47017984
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:318469896
mem_not_counted_for_evict:1073741824
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20967178
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions