[BUG] High RSS Memory Usage Even with Active Defrag #13612
Description
Describe the bug
Hello! We have been observing a strange issue since upgrading our redis standalone servers from 6.2.5 to 7.2.5. Occasionally, one of our redis servers will start growing in RSS memory until the system is running out of memory. We have activedefrag enabled and it starts running during this time but it doesn't seem to help. Can someone help take a look and see what is going on? I captured some data when this occurred last. If we need more data I will have to wait until it happens again to get back with it.
MEMORY STATS
1) "peak.allocated"
2) (integer) 9155310640
3) "total.allocated"
4) (integer) 9088238672
5) "startup.allocated"
6) (integer) 859496
7) "replication.backlog"
8) (integer) 0
9) "clients.slaves"
10) (integer) 0
11) "clients.normal"
12) (integer) 18185184
13) "cluster.links"
14) (integer) 0
15) "aof.buffer"
16) (integer) 0
17) "lua.caches"
18) (integer) 1416
19) "functions.caches"
20) (integer) 184
21) "db.0"
22) 1) "overhead.hashtable.main"
2) (integer) 547207768
3) "overhead.hashtable.expires"
4) (integer) 382011752
5) "overhead.hashtable.slot-to-keys"
6) (integer) 0
23) "overhead.total"
24) (integer) 948265800
25) "keys.count"
26) (integer) 10324751
27) "keys.bytes-per-key"
28) (integer) 880
29) "dataset.bytes"
30) (integer) 8139972872
31) "dataset.percentage"
32) "89.57447814941406"
33) "peak.percentage"
34) "99.26740264892578"
35) "allocator.allocated"
36) (integer) 9088373168
37) "allocator.active"
38) (integer) 12528521216
39) "allocator.resident"
40) (integer) 12667953152
41) "allocator-fragmentation.ratio"
42) "1.3785219192504883"
43) "allocator-fragmentation.bytes"
44) (integer) 3440148048
45) "allocator-rss.ratio"
46) "1.0111291408538818"
47) "allocator-rss.bytes"
48) (integer) 139431936
49) "rss-overhead.ratio"
50) "0.9998315572738647"
51) "rss-overhead.bytes"
52) (integer) -2134016
53) "fragmentation"
54) "1.393649697303772"
55) "fragmentation.bytes"
56) (integer) 3577581536
MALLOC-STATS
___ Begin jemalloc statistics ___
Version: "5.3.0-0-g0"
Build-time option settings
config.cache_oblivious: false
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.opt_safety_checks: false
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: false
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 8
opt.percpu_arena: "disabled"
opt.oversize_threshold: 8388608
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Arenas: 9
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 9088438136, active: 12537638912, metadata: 139436448 (n_thp 0), resident: 12677103616, mapped: 12690853888, retained: 1994452992
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 1, num_runs: 74773, run_interval: 9706952082 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 15920251 21 0 0 0 0 1 0 0 0 0 0
max_per_bg_thd 8447672 11 0 0 0 0 3 0 0 0 0 0
ctl 31840495 43 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
Merged arenas stats:
assigned threads: 1
uptime: 725830571650076
dss allocation precedence: "N/A"
decaying: time npages sweeps madvises purged
dirty: N/A 30 59808 593498 1893267
muzzy: N/A 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 8760041336 554242742 763 446073706 614 67182704643 92559 85734598 118 29525034 40
large: 328396800 1215150 1 1214086 1 11320091 15 1215150 1 859012 1
total: 9088438136 555457892 765 447287792 616 67194024734 92575 86949748 119 30384046 41
active: 12537638912
mapped: 12690853888
retained: 1994452992
base: 139338144
internal: 98304
metadata_thp: 0
tcache_bytes: 103760
tcache_stashed_bytes: 0
resident: 12677103616
abandoned_vm: 0
extent_avail: 1
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 15829375 21 0 0 0 0 2 0 0 0 0 0
extent_avail 17998331 24 0 0 2 0 104942 0 0 0 0 0
extents_dirty 22838844 31 0 0 58 0 119674 0 0 0 0 0
extents_muzzy 15864762 21 0 0 0 0 2 0 0 0 0 0
extents_retained 17626774 24 0 0 6 0 119698 0 0 0 0 0
decay_dirty 17250502 23 0 0 1 0 297186 0 0 0 0 0
decay_muzzy 15978550 22 0 0 0 0 296934 0 0 0 0 0
base 32529371 44 0 0 0 0 2 0 0 0 0 0
tcache_list 15829376 21 0 0 0 0 2 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5992 1460121 2 1459372 2 165227552 227 1 749 2 0 512 1 0.731 908665 1 751437 1 19 15835 0 17489496 24 0 0 0 0 2 0 0 0 0 0
16 1 165063952 53338498 73 43022001 59 19912092842 27433 1 10316497 42768 21276 256 1 0.942 10855872 14 1551039 2 78512 23817166 32 73137144 100 0 0 0 0 6578224 9 0 0 0 0
24 2 865151040 164045682 226 127997722 176 8437774623 11625 1 36047960 73048 55411 512 3 0.963 14373712 19 2039254 2 95160 65526118 90 198481414 273 0 0 0 0 24699694 34 0 0 0 0
32 3 328121216 47508388 65 37254600 51 11295660858 15562 1 10253788 85030 40353 128 1 0.942 6842004 9 1592562 2 158625 18131374 24 69605730 95 0 0 0 0 7114564 9 0 0 0 0
40 4 5440 534324 0 534188 0 2931833591 4039 1 136 1 0 512 5 0.265 521512 0 251277 0 1 0 0 16602165 22 0 0 0 0 2 0 0 0 0 0
48 5 109056 1985674 2 1983402 2 354711849 488 1 2272 21 15 256 3 0.422 1258241 1 1048496 1 353 817728 1 18185250 25 0 0 0 0 57184 0 0 0 0 0
56 6 905802688 71069706 97 54894658 75 1922065392 2648 1 16175048 33453 17199 512 7 0.944 9502332 13 1586140 2 54004 24549924 33 94025222 129 0 0 0 0 10067686 13 0 0 0 0
64 7 220889984 20255554 27 16804148 23 9163599655 12624 1 3451406 57263 27030 64 1 0.941 3259472 4 1498164 2 134042 7486391 10 37275666 51 0 0 0 0 2912754 4 0 0 0 0
80 8 68633040 10648106 14 9790193 13 4291824716 5912 1 857913 3689 3351 256 5 0.908 2192822 3 1504358 2 13474 4109182 5 26367533 36 0 0 0 0 1574130 2 0 0 0 0
96 9 17339232 2499640 3 2319023 3 859528950 1184 1 180617 1532 682 128 3 0.921 1000405 1 1121574 1 4790 1638823 2 18579568 25 0 0 0 0 370174 0 0 0 0 0
112 10 62473936 5700197 7 5142394 7 574210007 791 1 557803 2320 1654 256 7 0.939 2136234 2 1322845 1 5747 3710376 5 20741666 28 0 0 0 0 476220 0 0 0 0 0
128 11 80135040 18561135 25 17935080 24 337172693 464 1 626055 22087 19550 32 1 0.885 6767882 9 1566604 2 107037 12767896 17 32374751 44 0 0 0 0 1755476 2 0 0 0 0
160 12 2426561600 56990302 78 41824292 57 2579470684 3553 1 15166010 119315 44864 128 5 0.993 7166304 9 1513051 2 186127 23672807 32 88689395 122 0 0 0 0 12148526 16 0 0 0 0
192 13 1408278144 31675569 43 24340787 33 2049712618 2823 1 7334782 119689 58758 64 3 0.957 5002488 6 1479207 2 200693 12090507 16 52419929 72 0 0 0 0 6362682 8 0 0 0 0
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 60875629 83 0 0 0 0 1408028 1 0 0 0 0
256 15 20811776 1491703 2 1410407 1 44440621 61 1 81296 5296 1595 16 1 0.959 747211 1 776031 1 21314 1062821 1 17647301 24 0 0 0 0 228230 0 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 48252296 66 0 0 0 0 1153456 1 0 0 0 0
384 17 492847104 4230320 5 2946864 4 22604066 31 1 1283456 40108 0 32 3 1 1338847 1 1067491 1 67691 1704949 2 20171049 27 0 0 0 0 333398 0 0 0 0 0
448 18 13349056 1277906 1 1248109 1 3152267 4 1 29797 475 213 64 7 0.980 1040379 1 630996 0 1311 1153062 1 17614587 24 0 0 0 0 63762 0 0 0 0 0
512 19 40167936 1413530 1 1335077 1 3221982 4 1 78453 9822 119 8 1 0.998 968131 1 722665 0 27368 1087814 1 17760997 24 0 0 0 0 98412 0 0 0 0 0
640 20 116151680 1510154 2 1328667 1 3868401 5 1 181487 5672 0 32 5 0.999 984093 1 630889 0 11067 1151388 1 17813467 24 0 0 0 0 51560 0 0 0 0 0
768 21 130656000 800815 1 630690 0 2335944 3 1 170125 10633 0 16 3 0.999 507694 0 285925 0 17375 566498 0 16834672 23 0 0 0 0 25354 0 0 0 0 0
896 22 84373632 633803 0 539636 0 1370478 1 1 94167 2943 0 32 7 0.999 443954 0 367760 0 4943 224736 0 16705338 23 0 0 0 0 14328 0 0 0 0 0
1024 23 28119040 4382005 6 4354545 5 6779850 9 1 27460 6869 15 4 1 0.999 1908121 2 1613523 2 32073 4316967 5 19406739 26 0 0 0 0 13934 0 0 0 0 0
1280 24 4357120 68112 0 64708 0 78116 0 1 3404 213 1 16 5 0.998 61798 0 39568 0 408 52685 0 15937093 21 0 0 0 0 7270 0 0 0 0 0
1536 25 1015296 50198 0 49537 0 43369 0 1 661 87 20 8 3 0.949 38045 0 36189 0 342 38659 0 15906588 21 0 0 0 0 3350 0 0 0 0 0
1792 26 684544 41289 0 40907 0 53208 0 1 382 26 12 16 7 0.918 38242 0 29987 0 107 23204 0 15899656 21 0 0 0 0 2710 0 0 0 0 0
2048 27 1288192 4957078 6 4956449 6 6486777 8 1 629 319 9 2 1 0.985 1431194 1 1285085 1 360643 4650579 6 18909840 26 0 0 0 0 6118 0 0 0 0 0
2560 28 768000 14111 0 13811 0 13860 0 1 300 43 24 8 5 0.872 13441 0 10157 0 86 8192 0 15854369 21 0 0 0 0 1646 0 0 0 0 0
3072 29 439296 34197 0 34054 0 32242 0 1 143 40 13 4 3 0.893 31317 0 30906 0 153 9327 0 15892405 21 0 0 0 0 850 0 0 0 0 0
3584 30 340480 17907 0 17812 0 17699 0 1 95 13 2 8 7 0.913 17454 0 16204 0 55 3026 0 15863489 21 0 0 0 0 532 0 0 0 0 0
4096 31 1421312 461127 0 460780 0 376549 0 1 347 347 0 1 1 1 259944 0 344160 0 461127 0 0 16896310 23 0 0 0 0 3288 0 0 0 0 0
5120 32 378880 15199 0 15125 0 15139 0 1 74 21 5 4 5 0.880 14958 0 13920 0 85 2254 0 15858668 21 0 0 0 0 400 0 0 0 0 0
6144 33 344064 4787 0 4731 0 4623 0 1 56 31 5 2 3 0.903 4363 0 2799 0 467 3298 0 15837168 21 0 0 0 0 238 0 0 0 0 0
7168 34 157696 2176 0 2154 0 2143 0 1 22 7 4 4 7 0.785 1950 0 1161 0 110 1194 0 15832719 21 0 0 0 0 110 0 0 0 0 0
8192 35 1540096 21667 0 21479 0 18668 0 1 188 188 0 1 2 1 17908 0 19952 0 21667 0 0 15889896 21 0 0 0 0 1888 0 0 0 0 0
10240 36 276480 2210 0 2183 0 2255 0 1 27 16 4 2 5 0.843 2092 0 1407 0 253 1760 0 15833207 21 0 0 0 0 96 0 0 0 0 0
12288 37 110592 13676 0 13667 0 13732 0 1 9 9 0 1 3 1 13509 0 13024 0 13676 0 0 15869608 21 0 0 0 0 48 0 0 0 0 0
14336 38 71680 1510 0 1505 0 1605 0 1 5 4 2 2 7 0.625 1485 0 1105 0 282 813 0 15832266 21 0 0 0 0 34 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 1769472 316178 0 316070 0 364172 0 108
20480 40 15667200 832600 1 831835 1 10874836 14 765
24576 41 737280 5624 0 5594 0 7639 0 30
28672 42 802816 17034 0 17006 0 29725 0 28
32768 43 1277952 1189 0 1150 0 1194 0 39
40960 44 40960 248 0 247 0 248 0 1
49152 45 0 2541 0 2541 0 2541 0 0
57344 46 0 35906 0 35906 0 35906 0 0
65536 47 1900544 957 0 928 0 957 0 29
81920 48 81920 1249 0 1248 0 1249 0 1
98304 49 0 113 0 113 0 113 0 0
114688 50 0 301 0 301 0 301 0 0
131072 51 1835008 266 0 252 0 266 0 14
163840 52 0 298 0 298 0 298 0 0
196608 53 0 90 0 90 0 90 0 0
229376 54 0 125 0 125 0 125 0 0
262144 55 6553600 133 0 108 0 133 0 25
327680 56 327680 112 0 111 0 112 0 1
393216 57 0 50 0 50 0 50 0 0
458752 58 0 8 0 8 0 8 0 0
524288 59 5242880 43 0 33 0 43 0 10
655360 60 655360 8 0 7 0 8 0 1
786432 61 0 8 0 8 0 8 0 0
917504 62 0 20 0 20 0 20 0 0
1048576 63 2097152 18 0 16 0 18 0 2
---
2097152 67 12582912 12 0 6 0 12 0 6
---
4194304 71 8388608 5 0 3 0 5 0 2
---
8388608 75 0 3 0 3 0 3 0 0
---
16777216 79 0 3 0 3 0 3 0 0
---
33554432 83 0 3 0 3 0 3 0 0
---
67108864 87 0 3 0 3 0 3 0 0
---
134217728 91 268435456 2 0 0 0 2 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 10 40960 0 0 21144 86605824 21154 86646784
8192 1 10 81920 0 0 15494 126926848 15504 127008768
---
8388608 39 0 0 0 0 1 8388608 1 8388608
10485760 40 0 0 0 0 1 10485760 1 10485760
---
16777216 43 0 0 0 0 1 16777216 1 16777216
20971520 44 0 0 0 0 1 20971520 1 20971520
---
33554432 47 0 0 0 0 2 67108864 2 67108864
41943040 48 0 0 0 0 1 41943040 1 41943040
---
67108864 51 0 0 0 0 1 67108864 1 67108864
83886080 52 0 0 0 0 1 83886080 1 83886080
---
1342177280 68 0 0 0 0 1 1464250368 1 1464250368
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[0]:
assigned threads: 1
uptime: 725830571650076
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 30 59808 593486 1801107
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 8760041336 554242742 763 446073706 614 67182704643 92559 85734598 118 29525034 40
large: 59961344 1215136 1 1214074 1 11320077 15 1215136 1 859012 1
total: 8820002680 555457878 765 447287780 616 67194024720 92575 86949734 119 30384046 41
active: 12269203456
mapped: 12420321280
retained: 1677783040
base: 139244752
internal: 98304
metadata_thp: 0
tcache_bytes: 103760
tcache_stashed_bytes: 0
resident: 12408573952
abandoned_vm: 0
extent_avail: 0
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 7960125 10 0 0 0 0 1 0 0 0 0 0
extent_avail 10129062 13 0 0 2 0 104941 0 0 0 0 0
extents_dirty 14969568 20 0 0 58 0 119673 0 0 0 0 0
extents_muzzy 7995512 11 0 0 0 0 1 0 0 0 0 0
extents_retained 9757493 13 0 0 6 0 119697 0 0 0 0 0
decay_dirty 9234703 12 0 0 1 0 148985 0 0 0 0 0
decay_muzzy 8034898 11 0 0 0 0 148831 0 0 0 0 0
base 16790858 23 0 0 0 0 1 0 0 0 0 0
tcache_list 7960126 10 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5992 1460121 2 1459372 2 165227552 227 1 749 2 0 512 1 0.731 908665 1 751437 1 19 15835 0 9620246 13 0 0 0 0 1 0 0 0 0 0
16 1 165063952 53338498 73 43022001 59 19912092842 27433 1 10316497 42768 21276 256 1 0.942 10855872 14 1551039 2 78512 23817166 32 65267894 89 0 0 0 0 6578223 9 0 0 0 0
24 2 865151040 164045682 226 127997722 176 8437774623 11625 1 36047960 73048 55411 512 3 0.963 14373712 19 2039254 2 95160 65526118 90 190612164 262 0 0 0 0 24699693 34 0 0 0 0
32 3 328121216 47508388 65 37254600 51 11295660858 15562 1 10253788 85030 40353 128 1 0.942 6842004 9 1592562 2 158625 18131374 24 61736480 85 0 0 0 0 7114563 9 0 0 0 0
40 4 5440 534324 0 534188 0 2931833591 4039 1 136 1 0 512 5 0.265 521512 0 251277 0 1 0 0 8732915 12 0 0 0 0 1 0 0 0 0 0
48 5 109056 1985674 2 1983402 2 354711849 488 1 2272 21 15 256 3 0.422 1258241 1 1048496 1 353 817728 1 10316000 14 0 0 0 0 57183 0 0 0 0 0
56 6 905802688 71069706 97 54894658 75 1922065392 2648 1 16175048 33453 17199 512 7 0.944 9502332 13 1586140 2 54004 24549924 33 86155972 118 0 0 0 0 10067685 13 0 0 0 0
64 7 220889984 20255554 27 16804148 23 9163599655 12624 1 3451406 57263 27030 64 1 0.941 3259472 4 1498164 2 134042 7486391 10 29406416 40 0 0 0 0 2912753 4 0 0 0 0
80 8 68633040 10648106 14 9790193 13 4291824716 5912 1 857913 3689 3351 256 5 0.908 2192822 3 1504358 2 13474 4109182 5 18498283 25 0 0 0 0 1574129 2 0 0 0 0
96 9 17339232 2499640 3 2319023 3 859528950 1184 1 180617 1532 682 128 3 0.921 1000405 1 1121574 1 4790 1638823 2 10710318 14 0 0 0 0 370173 0 0 0 0 0
112 10 62473936 5700197 7 5142394 7 574210007 791 1 557803 2320 1654 256 7 0.939 2136234 2 1322845 1 5747 3710376 5 12872416 17 0 0 0 0 476219 0 0 0 0 0
128 11 80135040 18561135 25 17935080 24 337172693 464 1 626055 22087 19550 32 1 0.885 6767882 9 1566604 2 107037 12767896 17 24505501 33 0 0 0 0 1755475 2 0 0 0 0
160 12 2426561600 56990302 78 41824292 57 2579470684 3553 1 15166010 119315 44864 128 5 0.993 7166304 9 1513051 2 186127 23672807 32 80820145 111 0 0 0 0 12148525 16 0 0 0 0
192 13 1408278144 31675569 43 24340787 33 2049712618 2823 1 7334782 119689 58758 64 3 0.957 5002488 6 1479207 2 200693 12090507 16 44550679 61 0 0 0 0 6362681 8 0 0 0 0
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 53006379 73 0 0 0 0 1408027 1 0 0 0 0
256 15 20811776 1491703 2 1410407 1 44440621 61 1 81296 5296 1595 16 1 0.959 747211 1 776031 1 21314 1062821 1 9778051 13 0 0 0 0 228229 0 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 40383046 55 0 0 0 0 1153455 1 0 0 0 0
384 17 492847104 4230320 5 2946864 4 22604066 31 1 1283456 40108 0 32 3 1 1338847 1 1067491 1 67691 1704949 2 12301799 16 0 0 0 0 333397 0 0 0 0 0
448 18 13349056 1277906 1 1248109 1 3152267 4 1 29797 475 213 64 7 0.980 1040379 1 630996 0 1311 1153062 1 9745337 13 0 0 0 0 63761 0 0 0 0 0
512 19 40167936 1413530 1 1335077 1 3221982 4 1 78453 9822 119 8 1 0.998 968131 1 722665 0 27368 1087814 1 9891747 13 0 0 0 0 98411 0 0 0 0 0
640 20 116151680 1510154 2 1328667 1 3868401 5 1 181487 5672 0 32 5 0.999 984093 1 630889 0 11067 1151388 1 9944217 13 0 0 0 0 51559 0 0 0 0 0
768 21 130656000 800815 1 630690 0 2335944 3 1 170125 10633 0 16 3 0.999 507694 0 285925 0 17375 566498 0 8965422 12 0 0 0 0 25353 0 0 0 0 0
896 22 84373632 633803 0 539636 0 1370478 1 1 94167 2943 0 32 7 0.999 443954 0 367760 0 4943 224736 0 8836088 12 0 0 0 0 14327 0 0 0 0 0
1024 23 28119040 4382005 6 4354545 5 6779850 9 1 27460 6869 15 4 1 0.999 1908121 2 1613523 2 32073 4316967 5 11537489 15 0 0 0 0 13933 0 0 0 0 0
1280 24 4357120 68112 0 64708 0 78116 0 1 3404 213 1 16 5 0.998 61798 0 39568 0 408 52685 0 8067843 11 0 0 0 0 7269 0 0 0 0 0
1536 25 1015296 50198 0 49537 0 43369 0 1 661 87 20 8 3 0.949 38045 0 36189 0 342 38659 0 8037338 11 0 0 0 0 3349 0 0 0 0 0
1792 26 684544 41289 0 40907 0 53208 0 1 382 26 12 16 7 0.918 38242 0 29987 0 107 23204 0 8030406 11 0 0 0 0 2709 0 0 0 0 0
2048 27 1288192 4957078 6 4956449 6 6486777 8 1 629 319 9 2 1 0.985 1431194 1 1285085 1 360643 4650579 6 11040590 15 0 0 0 0 6117 0 0 0 0 0
2560 28 768000 14111 0 13811 0 13860 0 1 300 43 24 8 5 0.872 13441 0 10157 0 86 8192 0 7985119 11 0 0 0 0 1645 0 0 0 0 0
3072 29 439296 34197 0 34054 0 32242 0 1 143 40 13 4 3 0.893 31317 0 30906 0 153 9327 0 8023155 11 0 0 0 0 849 0 0 0 0 0
3584 30 340480 17907 0 17812 0 17699 0 1 95 13 2 8 7 0.913 17454 0 16204 0 55 3026 0 7994239 11 0 0 0 0 531 0 0 0 0 0
4096 31 1421312 461127 0 460780 0 376549 0 1 347 347 0 1 1 1 259944 0 344160 0 461127 0 0 9027060 12 0 0 0 0 3287 0 0 0 0 0
5120 32 378880 15199 0 15125 0 15139 0 1 74 21 5 4 5 0.880 14958 0 13920 0 85 2254 0 7989418 11 0 0 0 0 399 0 0 0 0 0
6144 33 344064 4787 0 4731 0 4623 0 1 56 31 5 2 3 0.903 4363 0 2799 0 467 3298 0 7967918 10 0 0 0 0 237 0 0 0 0 0
7168 34 157696 2176 0 2154 0 2143 0 1 22 7 4 4 7 0.785 1950 0 1161 0 110 1194 0 7963469 10 0 0 0 0 109 0 0 0 0 0
8192 35 1540096 21667 0 21479 0 18668 0 1 188 188 0 1 2 1 17908 0 19952 0 21667 0 0 8020646 11 0 0 0 0 1887 0 0 0 0 0
10240 36 276480 2210 0 2183 0 2255 0 1 27 16 4 2 5 0.843 2092 0 1407 0 253 1760 0 7963957 10 0 0 0 0 95 0 0 0 0 0
12288 37 110592 13676 0 13667 0 13732 0 1 9 9 0 1 3 1 13509 0 13024 0 13676 0 0 8000358 11 0 0 0 0 47 0 0 0 0 0
14336 38 71680 1510 0 1505 0 1605 0 1 5 4 2 2 7 0.625 1485 0 1105 0 282 813 0 7963016 10 0 0 0 0 33 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 1769472 316178 0 316070 0 364172 0 108
20480 40 15667200 832600 1 831835 1 10874836 14 765
24576 41 737280 5624 0 5594 0 7639 0 30
28672 42 802816 17034 0 17006 0 29725 0 28
32768 43 1277952 1189 0 1150 0 1194 0 39
40960 44 40960 248 0 247 0 248 0 1
49152 45 0 2541 0 2541 0 2541 0 0
57344 46 0 35906 0 35906 0 35906 0 0
65536 47 1900544 957 0 928 0 957 0 29
81920 48 81920 1249 0 1248 0 1249 0 1
98304 49 0 113 0 113 0 113 0 0
114688 50 0 301 0 301 0 301 0 0
131072 51 1835008 266 0 252 0 266 0 14
163840 52 0 298 0 298 0 298 0 0
196608 53 0 90 0 90 0 90 0 0
229376 54 0 125 0 125 0 125 0 0
262144 55 6553600 133 0 108 0 133 0 25
327680 56 327680 112 0 111 0 112 0 1
393216 57 0 50 0 50 0 50 0 0
458752 58 0 8 0 8 0 8 0 0
524288 59 5242880 43 0 33 0 43 0 10
655360 60 655360 8 0 7 0 8 0 1
786432 61 0 8 0 8 0 8 0 0
917504 62 0 20 0 20 0 20 0 0
1048576 63 2097152 18 0 16 0 18 0 2
---
2097152 67 12582912 12 0 6 0 12 0 6
---
4194304 71 8388608 5 0 3 0 5 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 10 40960 0 0 21144 86605824 21154 86646784
8192 1 10 81920 0 0 15494 126926848 15504 127008768
---
1342177280 68 0 0 0 0 1 1464250368 1 1464250368
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[8]:
assigned threads: 0
uptime: 717554192152229
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 0 0 0 12 92160
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 0 0 0 0 0 0 0 0 0 0 0
large: 268435456 14 0 12 0 14 0 14 0 0 0
total: 268435456 14 0 12 0 14 0 14 0 0 0
active: 268435456
mapped: 270532608
retained: 316669952
base: 93392
internal: 0
metadata_thp: 0
tcache_bytes: 0
tcache_stashed_bytes: 0
resident: 268529664
abandoned_vm: 0
extent_avail: 1
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 7869250 10 0 0 0 0 1 0 0 0 0 0
extent_avail 7869269 10 0 0 0 0 1 0 0 0 0 0
extents_dirty 7869276 10 0 0 0 0 1 0 0 0 0 0
extents_muzzy 7869250 10 0 0 0 0 1 0 0 0 0 0
extents_retained 7869281 10 0 0 0 0 1 0 0 0 0 0
decay_dirty 8015799 11 0 0 0 0 148201 0 0 0 0 0
decay_muzzy 7943652 11 0 0 0 0 148103 0 0 0 0 0
base 15738513 21 0 0 0 0 1 0 0 0 0 0
tcache_list 7869250 10 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
---
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
---
8388608 75 0 3 0 3 0 3 0 0
---
16777216 79 0 3 0 3 0 3 0 0
---
33554432 83 0 3 0 3 0 3 0 0
---
67108864 87 0 3 0 3 0 3 0 0
---
134217728 91 268435456 2 0 0 0 2 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
---
8388608 39 0 0 0 0 1 8388608 1 8388608
10485760 40 0 0 0 0 1 10485760 1 10485760
---
16777216 43 0 0 0 0 1 16777216 1 16777216
20971520 44 0 0 0 0 1 20971520 1 20971520
---
33554432 47 0 0 0 0 2 67108864 2 67108864
41943040 48 0 0 0 0 1 41943040 1 41943040
---
67108864 51 0 0 0 0 1 67108864 1 67108864
83886080 52 0 0 0 0 1 83886080 1 83886080
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
--- End jemalloc statistics ---
To reproduce
Unsure what causes this issue to reproduce.
Expected behavior
I expect active defrag running to help with external memory fragmentation