Skip to content

[CRASH] Assertion failed: pthread_mutex_unlock #997

Description

@towest-nr

Crash report

...
1:S 05 Sep 2024 23:26:20.992 * RDB memory usage when created 2198.31 Mb
1:S 05 Sep 2024 23:26:27.858 * Done loading RDB, keys loaded: 1022610, keys expired: 0.
1:S 05 Sep 2024 23:26:27.858 * PRIMARY <-> REPLICA sync: Finished with success

=== VALKEY BUG REPORT START: Cut & paste starting from here ===
1:S 05 Sep 2024 23:29:26.128 # === ASSERTION FAILED ===
1:S 05 Sep 2024 23:29:26.128 # ==> ae.c:205 'pthread_mutex_unlock(&(eventLoop)->poll_mutex) == 0' is not true

------ STACK TRACE ------

1 redis-server
redis-server *redacted*:6379 [cluster](afterSleep+0x2e)[0x556ad98eff5e]
redis-server *redacted*:6379 [cluster](aeMain+0x64)[0x556ad98e86b4]
redis-server *redacted*:6379 [cluster](main+0x52f)[0x556ad98dc79f]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f1b00996d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f1b00996e40]
redis-server *redacted*:6379 [cluster](_start+0x25)[0x556ad98dce95]

15 bio_close_file
/lib/x86_64-linux-gnu/libc.so.6(+0x91117)[0x7f1b009fe117]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x211)[0x7f1b00a00a41]
redis-server *redacted*:6379 [cluster](bioProcessBackgroundJobs+0x1ff)[0x556ad99bf82f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f1b00a01ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f1b00a93850]

18 io_thd_1 *
redis-server *redacted*:6379 [cluster](aeCreateFileEvent+0x146)[0x556ad98e1d76]
redis-server *redacted*:6379 [cluster](+0x1f5d56)[0x556ad9a48d56]
redis-server *redacted*:6379 [cluster](+0x1f5ee4)[0x556ad9a48ee4]
redis-server *redacted*:6379 [cluster](_writeToClient+0x357)[0x556ad99182d7]
redis-server *redacted*:6379 [cluster](ioThreadWriteToClient+0x2a)[0x556ad99274aa]
redis-server *redacted*:6379 [cluster](+0xc1acb)[0x556ad9914acb]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f1b00a01ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f1b00a93850]

19 io_thd_2
/lib/x86_64-linux-gnu/libc.so.6(+0x912c0)[0x7f1b009fe2c0]
/lib/x86_64-linux-gnu/libc.so.6(pthread_mutex_lock+0x112)[0x7f1b00a05002]
redis-server *redacted*:6379 [cluster](+0xc1b46)[0x556ad9914b46]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f1b00a01ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f1b00a93850]

17 bio_lazy_free
/lib/x86_64-linux-gnu/libc.so.6(+0x91117)[0x7f1b009fe117]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x211)[0x7f1b00a00a41]
redis-server *redacted*:6379 [cluster](bioProcessBackgroundJobs+0x1ff)[0x556ad99bf82f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f1b00a01ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f1b00a93850]

16 bio_aof
/lib/x86_64-linux-gnu/libc.so.6(+0x91117)[0x7f1b009fe117]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x211)[0x7f1b00a00a41]
redis-server *redacted*:6379 [cluster](bioProcessBackgroundJobs+0x1ff)[0x556ad99bf82f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f1b00a01ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f1b00a93850]

6/6 expected stacktraces.

------ STACK TRACE DONE ------

------ INFO OUTPUT ------
# Server
redis_version:7.2.4
server_name:valkey
valkey_version:7.9.240
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7054fc2b9af49195
server_mode:cluster
os:Linux 5.10.210-201.855.amzn2.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
gcc_version:11.4.0
process_id:1
process_supervised:no
run_id:c1954153d0077bcc1c25629778754132b8b43f35
tcp_port:6379
server_time_usec:1725578966136331
uptime_in_seconds:203
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:14302934
executable:/data/redis-server
config_file:/etc/valkey/valkey.conf
io_threads_active:1
availability_zone:
listener2:name=tls,bind=*redacted*,bind=::,port=6379

# Clients
connected_clients:62
cluster_connections:6
maxclients:65000
client_recent_max_input_buffer:57344
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:2268809744
used_memory_human:2.11G
used_memory_rss:2382929920
used_memory_rss_human:2.22G
used_memory_peak:2296366144
used_memory_peak_human:2.14G
used_memory_peak_perc:98.80%
used_memory_overhead:116946172
used_memory_startup:5697352
used_memory_dataset:2151863572
used_memory_dataset_perc:95.08%
allocator_allocated:2273572464
allocator_active:2331004928
allocator_resident:2377592832
allocator_muzzy:0
total_system_memory:133518946304
total_system_memory_human:124.35G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:33792
used_memory_vm_total:65536
used_memory_vm_total_human:64.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:12288000000
maxmemory_human:11.44G
maxmemory_policy:noeviction
allocator_frag_ratio:1.03
allocator_frag_bytes:57432464
allocator_rss_ratio:1.02
allocator_rss_bytes:46587904
rss_overhead_ratio:1.00
rss_overhead_bytes:5337088
mem_fragmentation_ratio:1.05
mem_fragmentation_bytes:111747536
mem_not_counted_for_evict:12248
mem_replication_backlog:10539172
mem_total_replication_buffers:10539032
mem_clients_slaves:0
mem_clients_normal:2257152
mem_cluster_links:6432
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:2

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:7350588
rdb_bgsave_in_progress:0
rdb_last_save_time:1725578763
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:1022610
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:315
total_commands_processed:9831979
instantaneous_ops_per_sec:56546
total_net_input_bytes:6449079074
total_net_output_bytes:3282501046
total_net_repl_input_bytes:6223790676
total_net_repl_output_bytes:0
instantaneous_input_kbps:22947.52
instantaneous_output_kbps:23266.48
instantaneous_input_repl_kbps:21306.95
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:1709944
keyspace_misses:770015
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:50414
dump_payload_sanitizations:0
total_reads_processed:7063789
total_writes_processed:2531645
io_threaded_reads_processed:1011038
io_threaded_writes_processed:1060736
io_threaded_freed_objects:0
io_threaded_poll_processed:346554
io_threaded_total_prefetch_batches:57038
io_threaded_total_prefetch_entries:291266
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:913
reply_buffer_expands:1822
eventloop_cycles:18322963
eventloop_duration_sum:110546139
eventloop_duration_cmd_sum:17855051
instantaneous_eventloop_cycles_per_sec:144147
instantaneous_eventloop_duration_usec:9
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:slave
master_host:*redacted*
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:290321071943
slave_repl_offset:290321071943
replicas_repl_buffer_size:0
replicas_repl_buffer_peak:0
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
replicas_waiting_psync:0
master_failover_state:no-failover
master_replid:042493f11c982d5142033a34d05a4b9e8c9e1917
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:290321071943
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:10485760
repl_backlog_first_byte_offset:290310568649
repl_backlog_histlen:10503295

# CPU
used_cpu_sys:59.925226
used_cpu_user:105.671500
used_cpu_sys_children:0.002204
used_cpu_user_children:0.006241
used_cpu_sys_main_thread:8.502897
used_cpu_user_main_thread:22.038012

# Modules

# Commandstats
cmdstat_cluster|slots:calls=6,usec=80,usec_per_call=13.33,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=6,usec=301,usec_per_call=50.17,rejected_calls=0,failed_calls=0
cmdstat_cluster|nodes:calls=202,usec=12117,usec_per_call=59.99,rejected_calls=0,failed_calls=0
cmdstat_info:calls=129,usec=8196,usec_per_call=63.53,rejected_calls=0,failed_calls=0
cmdstat_unlink:calls=2041259,usec=4053552,usec_per_call=1.99,rejected_calls=0,failed_calls=0
cmdstat_pexpireat:calls=2654522,usec=4447420,usec_per_call=1.68,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=6,usec=7,usec_per_call=1.17,rejected_calls=0,failed_calls=0
cmdstat_latency|histogram:calls=6,usec=1717,usec_per_call=286.17,rejected_calls=0,failed_calls=0
cmdstat_lpush:calls=2654807,usec=5855675,usec_per_call=2.21,rejected_calls=0,failed_calls=0
cmdstat_lrange:calls=2479959,usec=3472673,usec_per_call=1.40,rejected_calls=50414,failed_calls=0
cmdstat_readonly:calls=91,usec=30,usec_per_call=0.33,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=84,usec=277,usec_per_call=3.30,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=61,usec=7,usec_per_call=0.11,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=231,usec=1335,usec_per_call=5.78,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=6,usec=1317,usec_per_call=219.50,rejected_calls=0,failed_calls=0
cmdstat_select:calls=1,usec=0,usec_per_call=0.00,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=129,usec=154,usec_per_call=1.19,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=462,usec=185,usec_per_call=0.40,rejected_calls=0,failed_calls=0
cmdstat_slowlog|len:calls=6,usec=5,usec_per_call=0.83,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=6,usec=3,usec_per_call=0.50,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_LOADING:count=50414

# Latencystats
latency_percentiles_usec_cluster|slots:p50=6.015,p99=49.151,p99.9=49.151
latency_percentiles_usec_cluster|info:p50=43.007,p99=71.167,p99.9=71.167
latency_percentiles_usec_cluster|nodes:p50=60.159,p99=92.159,p99.9=95.231
latency_percentiles_usec_info:p50=57.087,p99=205.823,p99.9=207.871
latency_percentiles_usec_unlink:p50=2.007,p99=5.023,p99.9=14.015
latency_percentiles_usec_pexpireat:p50=2.007,p99=4.015,p99.9=18.047
latency_percentiles_usec_latency|latest:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_latency|histogram:p50=292.863,p99=311.295,p99.9=311.295
latency_percentiles_usec_lpush:p50=2.007,p99=10.047,p99.9=28.031
latency_percentiles_usec_lrange:p50=1.003,p99=4.015,p99.9=16.063
latency_percentiles_usec_readonly:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_auth:p50=3.007,p99=8.031,p99.9=11.007
latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_hello:p50=6.015,p99=11.007,p99.9=14.015
latency_percentiles_usec_config|get:p50=224.255,p99=227.327,p99.9=227.327
latency_percentiles_usec_select:p50=0.001,p99=0.001,p99.9=0.001
latency_percentiles_usec_client|setname:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_client|setinfo:p50=0.001,p99=1.003,p99.9=2.007
latency_percentiles_usec_slowlog|len:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_slowlog|get:p50=0.001,p99=1.003,p99.9=1.003

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=1013033,expires=1012810,avg_ttl=0

# Cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:4
cluster_size:2
cluster_current_epoch:6
cluster_my_epoch:5
cluster_stats_messages_ping_sent:251
cluster_stats_messages_pong_sent:256
cluster_stats_messages_sent:507
cluster_stats_messages_ping_received:256
cluster_stats_messages_pong_received:251
cluster_stats_messages_received:507
total_cluster_links_buffer_limit_exceeded:0

------ CLUSTER NODES OUTPUT ------
537c00be765db05a7c00b03fdbc149b2ffd728b0 *redacted*:0@16379,follower-1.redis.default.svc.cluster.local,tls-port=6379,shard-id=1d2a9e8aa5b4937c47c48a96aa294da2f77a68bd myself,slave f90d51f138c86ee0bada66afc89e2916da8ab849 0 0 5 connected
cfe5e107fce3bca937f4c26e96fd8576b2db5c11 *redacted*:0@16379,leader-0.redis.default.svc.cluster.local,tls-port=6379,shard-id=646e878d04ee2d406170effc45d52e80d2d65d5a master - 0 1725578966096 4 connected 0-5460 10923-13653
ba740172376b6be343d28079741a183d83d2fc5b *redacted*:0@16379,follower-0.redis.default.svc.cluster.local,tls-port=6379,shard-id=646e878d04ee2d406170effc45d52e80d2d65d5a slave cfe5e107fce3bca937f4c26e96fd8576b2db5c11 0 1725578965000 4 connected
f90d51f138c86ee0bada66afc89e2916da8ab849 *redacted*:0@16379,leader-1.redis.default.svc.cluster.local,tls-port=6379,shard-id=1d2a9e8aa5b4937c47c48a96aa294da2f77a68bd master - 0 1725578965995 5 connected 5461-10922 13654-16383

------ CLIENT LIST OUTPUT ------
id=37 addr=*redacted*:48888 laddr=*redacted*:6379 fd=38 name=*redacted* age=194 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3137506 tot-net-out=36130122 tot-cmds=34485
id=32 addr=*redacted*:48816 laddr=*redacted*:6379 fd=34 name=*redacted* age=195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3155965 tot-net-out=22347422 tot-cmds=34631
id=73 addr=*redacted*:44274 laddr=*redacted*:6379 fd=65 name=*redacted* age=189 idle=1 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3043865 tot-net-out=42981068 tot-cmds=33456
id=102 addr=*redacted*:38686 laddr=*redacted*:6379 fd=85 name=*redacted* age=187 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=6144 rbp=5829 obl=0 oll=0 omem=0 tot-mem=27520 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3043449 tot-net-out=34990268 tot-cmds=33412
id=103 addr=*redacted*:45558 laddr=*redacted*:6379 fd=86 name=*redacted* age=187 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=4 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3018324 tot-net-out=20459459 tot-cmds=33159
id=92 addr=*redacted*:56124 laddr=*redacted*:6379 fd=79 name=*redacted* age=188 idle=1 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=8192 rbp=8192 obl=0 oll=0 omem=0 tot-mem=29568 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3027368 tot-net-out=54934271 tot-cmds=33241
id=74 addr=*redacted*:57912 laddr=*redacted*:6379 fd=66 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3040500 tot-net-out=54796001 tot-cmds=33451
id=60 addr=*redacted*:46478 laddr=*redacted*:6379 fd=57 name=*redacted* age=190 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3091499 tot-net-out=64407292 tot-cmds=33959
id=155 addr=*redacted*:53600 laddr=*redacted*:6379 fd=95 name=*redacted* age=134 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2289318 tot-net-out=20772415 tot-cmds=25723
id=6 addr=*redacted*:37212 laddr=*redacted*:6379 fd=21 name=*redacted* age=197 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3168106 tot-net-out=43063117 tot-cmds=34811
id=76 addr=*redacted*:40078 laddr=*redacted*:6379 fd=68 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3030084 tot-net-out=42833620 tot-cmds=33241
id=48 addr=*redacted*:57556 laddr=*redacted*:6379 fd=48 name=*redacted* age=192 idle=1 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3084547 tot-net-out=62930328 tot-cmds=33923
id=181 addr=*redacted*:59672 laddr=*redacted*:6379 fd=106 name=*redacted* age=128 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=12288 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2212156 tot-net-out=32749295 tot-cmds=24856
id=57 addr=*redacted*:38438 laddr=*redacted*:6379 fd=55 name=*redacted* age=190 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3065855 tot-net-out=35462222 tot-cmds=33707
id=26 addr=*redacted*:60770 laddr=*redacted*:6379 fd=30 name=*redacted* age=196 idle=1 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=14793 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3145578 tot-net-out=47946064 tot-cmds=34512
id=31 addr=*redacted*:42814 laddr=*redacted*:6379 fd=33 name=*redacted* age=195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3142650 tot-net-out=42654094 tot-cmds=34540
id=150 addr=*redacted*:51524 laddr=*redacted*:6379 fd=91 name=*redacted* age=135 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2290793 tot-net-out=36121055 tot-cmds=25739
id=50 addr=*redacted*:56150 laddr=*redacted*:6379 fd=50 name=*redacted* age=192 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3079145 tot-net-out=42639568 tot-cmds=33822
id=79 addr=*redacted*:53928 laddr=*redacted*:6379 fd=70 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3079101 tot-net-out=56517765 tot-cmds=33890
id=28 addr=*redacted*:34376 laddr=*redacted*:6379 fd=31 name=*redacted* age=195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=14336 rbp=14336 obl=0 oll=0 omem=0 tot-mem=35712 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3141954 tot-net-out=42278210 tot-cmds=34576
id=183 addr=*redacted*:43196 laddr=*redacted*:6379 fd=108 name=*redacted* age=128 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=8192 rbp=8192 obl=0 oll=0 omem=0 tot-mem=29568 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2213793 tot-net-out=39802701 tot-cmds=24874
id=160 addr=*redacted*:47452 laddr=*redacted*:6379 fd=96 name=*redacted* age=132 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2279947 tot-net-out=48486699 tot-cmds=25617
id=96 addr=*redacted*:33890 laddr=*redacted*:6379 fd=80 name=*redacted* age=188 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=5086 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3070066 tot-net-out=36079364 tot-cmds=33756
id=61 addr=*redacted*:59792 laddr=*redacted*:6379 fd=58 name=*redacted* age=190 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3048483 tot-net-out=15223583 tot-cmds=33502
id=62 addr=*redacted*:48410 laddr=*redacted*:6379 fd=59 name=*redacted* age=190 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=6144 rbp=6144 obl=0 oll=0 omem=0 tot-mem=27520 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3043894 tot-net-out=55459576 tot-cmds=33486
id=63 addr=*redacted*:59162 laddr=*redacted*:6379 fd=60 name=*redacted* age=190 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3101058 tot-net-out=28052609 tot-cmds=34042
id=81 addr=*redacted*:42326 laddr=*redacted*:6379 fd=69 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3069290 tot-net-out=7919467 tot-cmds=33694
id=82 addr=*redacted*:57238 laddr=*redacted*:6379 fd=71 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3068376 tot-net-out=62848849 tot-cmds=33695
id=137 addr=*redacted*:35408 laddr=*redacted*:6379 fd=87 name=*redacted* age=136 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2325135 tot-net-out=49646819 tot-cmds=26125
id=46 addr=*redacted*:36400 laddr=*redacted*:6379 fd=46 name=*redacted* age=193 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3145231 tot-net-out=64080127 tot-cmds=34598
id=47 addr=*redacted*:47122 laddr=*redacted*:6379 fd=47 name=*redacted* age=192 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3125709 tot-net-out=43767323 tot-cmds=34330
id=151 addr=*redacted*:47890 laddr=*redacted*:6379 fd=92 name=*redacted* age=135 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=14336 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2317248 tot-net-out=41454780 tot-cmds=26037
id=9 addr=*redacted*:60632 laddr=*redacted*:6379 fd=22 name=*redacted* age=196 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3157125 tot-net-out=35106467 tot-cmds=34780
id=10 addr=*redacted*:55336 laddr=*redacted*:6379 fd=23 name=*redacted* age=196 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3169314 tot-net-out=55483412 tot-cmds=34860
id=84 addr=*redacted*:54196 laddr=*redacted*:6379 fd=73 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3064024 tot-net-out=56007559 tot-cmds=33653
id=113 addr=*redacted*:6379 laddr=*redacted*:39246 fd=16 name=*redacted* age=179 idle=0 flags=M db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=57334 argv-mem=0 multi-mem=0 rbs=1024 rbp=46 obl=0 oll=0 omem=0 tot-mem=59264 events=r cmd=pexpireat user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=4480734999 tot-net-out=8188 tot-cmds=7350609
id=85 addr=*redacted*:53040 laddr=*redacted*:6379 fd=74 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=7168 rbp=6317 obl=0 oll=0 omem=0 tot-mem=28544 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3049256 tot-net-out=7885062 tot-cmds=33557
id=13 addr=*redacted*:59550 laddr=*redacted*:6379 fd=24 name=*redacted* age=196 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3183344 tot-net-out=63932230 tot-cmds=35001
id=51 addr=*redacted*:42636 laddr=*redacted*:6379 fd=51 name=*redacted* age=191 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=8192 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3116552 tot-net-out=48032286 tot-cmds=34271
id=97 addr=*redacted*:42912 laddr=*redacted*:6379 fd=81 name=*redacted* age=188 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=4 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=1717152 tot-net-out=20194497 tot-cmds=18681
id=174 addr=*redacted*:41340 laddr=*redacted*:6379 fd=100 name=*redacted* age=128 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2170097 tot-net-out=46480873 tot-cmds=24383
id=176 addr=*redacted*:52512 laddr=*redacted*:6379 fd=102 name=*redacted* age=128 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2205567 tot-net-out=40854676 tot-cmds=24782
id=53 addr=*redacted*:47162 laddr=*redacted*:6379 fd=52 name=*redacted* age=191 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=15679 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3121847 tot-net-out=49238876 tot-cmds=34283
id=164 addr=*redacted*:55704 laddr=*redacted*:6379 fd=97 name=*redacted* age=131 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=8504 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2240852 tot-net-out=15218033 tot-cmds=25178
id=14 addr=*redacted*:39804 laddr=*redacted*:6379 fd=25 name=*redacted* age=196 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3103435 tot-net-out=54897782 tot-cmds=34086
id=29 addr=*redacted*:50746 laddr=*redacted*:6379 fd=32 name=*redacted* age=195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3152889 tot-net-out=63653670 tot-cmds=34649
id=100 addr=*redacted*:50752 laddr=*redacted*:6379 fd=83 name=*redacted* age=187 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=12894 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3030943 tot-net-out=15032807 tot-cmds=33232
id=88 addr=*redacted*:50512 laddr=*redacted*:6379 fd=76 name=*redacted* age=188 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3053602 tot-net-out=62911463 tot-cmds=33489
id=89 addr=*redacted*:41232 laddr=*redacted*:6379 fd=77 name=*redacted* age=188 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=10240 rbp=5781 obl=0 oll=0 omem=0 tot-mem=31616 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3056051 tot-net-out=27257585 tot-cmds=33521
id=34 addr=*redacted*:49150 laddr=*redacted*:6379 fd=36 name=*redacted* age=195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=3 omem=69704 tot-mem=107464 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3144688 tot-net-out=42738696 tot-cmds=34545
id=66 addr=*redacted*:50124 laddr=*redacted*:6379 fd=63 name=*redacted* age=189 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3067177 tot-net-out=63125755 tot-cmds=33673
id=35 addr=*redacted*:45542 laddr=*redacted*:6379 fd=37 name=*redacted* age=195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3142327 tot-net-out=35719060 tot-cmds=34513
id=101 addr=*redacted*:35356 laddr=*redacted*:6379 fd=84 name=*redacted* age=187 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3043683 tot-net-out=48130353 tot-cmds=33399
id=178 addr=*redacted*:48190 laddr=*redacted*:6379 fd=104 name=*redacted* age=128 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2188386 tot-net-out=46340477 tot-cmds=24589
id=144 addr=*redacted*:59456 laddr=*redacted*:6379 fd=88 name=*redacted* age=136 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=6182 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2308199 tot-net-out=35426388 tot-cmds=25935
id=153 addr=*redacted*:54384 laddr=*redacted*:6379 fd=94 name=*redacted* age=135 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=12288 rbp=6144 obl=0 oll=0 omem=0 tot-mem=33664 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=1928057 tot-net-out=20961908 tot-cmds=21664
id=54 addr=*redacted*:34302 laddr=*redacted*:6379 fd=53 name=*redacted* age=191 idle=1 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3080879 tot-net-out=63958072 tot-cmds=33886
id=188 addr=*redacted*:44852 laddr=*redacted*:6379 fd=110 name=*redacted* age=127 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2165237 tot-net-out=31505422 tot-cmds=24328
id=180 addr=*redacted*:35718 laddr=*redacted*:6379 fd=105 name=*redacted* age=128 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=4096 rbp=2048 obl=0 oll=0 omem=0 tot-mem=25472 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2176635 tot-net-out=19301963 tot-cmds=24457
id=18 addr=*redacted*:37316 laddr=*redacted*:6379 fd=26 name=*redacted* age=196 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=12288 rbp=8162 obl=0 oll=0 omem=0 tot-mem=33664 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=1753439 tot-net-out=27013474 tot-cmds=19155
id=147 addr=*redacted*:34124 laddr=*redacted*:6379 fd=89 name=*redacted* age=136 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=2304347 tot-net-out=41962047 tot-cmds=25892
id=44 addr=*redacted*:41866 laddr=*redacted*:6379 fd=45 name=*redacted* age=193 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=lrange user=*redacted* redir=-1 resp=3 lib-name=Lettuce lib-ver=6.3.2.RELEASE/8941aea tot-net-in=3122758 tot-net-out=35192063 tot-cmds=34379

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
io-threads-do-reads yes
io-threads 3
dual-channel-replication-enabled no
activedefrag no
repl-diskless-sync yes
debug-context ""
lazyfree-lazy-eviction yes
lazyfree-lazy-server-del yes
list-compress-depth 0
lazyfree-lazy-user-flush yes
slave-read-only yes
repl-diskless-load disabled
client-query-buffer-limit 1gb
sanitize-dump-payload no
proto-max-bulk-len 512mb
lazyfree-lazy-user-del yes
lazyfree-lazy-expire yes
replica-read-only yes

------ FAST MEMORY TEST ------
1:S 05 Sep 2024 23:29:26.137 # main thread terminated
1:S 05 Sep 2024 23:29:26.137 # Bio worker thread #0 terminated
1:S 05 Sep 2024 23:29:26.137 # Bio worker thread #1 terminated
1:S 05 Sep 2024 23:29:26.137 # Bio worker thread #2 terminated
1:S 05 Sep 2024 23:29:26.137 * IO thread(tid:139753877386816) terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== VALKEY BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/valkey-io/valkey/issues

  If a module was involved, please open in the module's repo instead.

  Suspect RAM error? Use valkey-server --test-memory to verify it.

  Some other issues could be detected by valkey-server --check-system

Additional information

I am in the process of doing a trial run with Valkey in one of our staging environments to determine if it's viable for us given our scale and heavy read/write load. I decided to run with 8.0.0 rc2 as a starting point and hit this issue yesterday evening. Since then it's been running without any problems but I will continue to monitor.

We're running in cluster mode with 2 shards, each primary with a single replica. We're pushing somewhere around 2-3GB/min into each primary and the same being read back out from the replica. Unfortunately I don't have any steps to reproduce at this point but if I find any other evidence that might help, I'll be sure to add it.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions