Skip to content

[CRASH] Guru Meditation: Unknown btype in replyToBlockedClientTimedOut() #605

@egeneralov

Description

@egeneralov

Crash report

Hello, inside container i see next command: keydb-server /etc/keydb/redis.conf --active-replica yes --multi-master yes --appendonly no --bind 0.0.0.0 --port 6379 --protected-mode no --server-threads 2 --masterauth AA000000aaaaaaa --requirepass AA000000aaaaaaa

after connecting clients has an error:

1:1:C 14 Mar 2023 10:18:24.903 * Notice: "active-replica yes" implies "replica-read-only no"
1:1:C 14 Mar 2023 10:18:24.904 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1:1:C 14 Mar 2023 10:18:24.904 # KeyDB version=6.3.2, bits=64, commit=d1dff8c2, modified=1, pid=1, just started
1:1:C 14 Mar 2023 10:18:24.904 # Configuration loaded
1:1:M 14 Mar 2023 10:18:24.904 * monotonic clock: POSIX clock_gettime
                                                                      
                  _                                                   
               _-(+)-_                                                
            _-- /   \ --_                                            
         _--   /     \   --_            KeyDB  6.3.2 (d1dff8c2/1) 64 bit     
     __--     /       \     --__                                     
    (+) _    /         \    _ (+)       Running in standalone mode
     |   -- /           \ --   |        Port: 6379
     |     /--_   _   _--\     |        PID: 1
     |    /     -(+)-     \    |                                     
     |   /        |        \   |        https://docs.keydb.dev       
     |  /         |         \  |                                     
     | /          |          \ |                                     
    (+)_ -- -- -- | -- -- -- _(+)                                     
        --_       |       _--                                         
            --_   |   _--                                             
                -(+)-        KeyDB has now joined Snap! See the announcement at:  https://docs.keydb.dev/news

                                                                     
1:1:M 14 Mar 2023 10:18:25.193 # Server initialized
1:16:M 14 Mar 2023 10:18:25.194 * Thread 0 alive.
1:17:M 14 Mar 2023 10:18:25.194 * Thread 1 alive.


=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:16:M 14 Mar 2023 10:19:20.006 # ------------------------------------------------
1:16:M 14 Mar 2023 10:19:20.006 # !!! Software Failure. Press left mouse button to continue
1:16:M 14 Mar 2023 10:19:20.006 # Guru Meditation: Unknown btype in replyToBlockedClientTimedOut(). #blocked.cpp:246

------ STACK TRACE ------

Backtrace:
keydb-server 0.0.0.0:6379(replyToBlockedClientTimedOut(client*)+0x72) [0x5561e31e35b2]
keydb-server 0.0.0.0:6379(checkBlockedClientTimeout(client*, long long)+0x49) [0x5561e31ee5b9]
keydb-server 0.0.0.0:6379(handleBlockedClientsTimeout()+0x127) [0x5561e31ee6f7]
keydb-server 0.0.0.0:6379(beforeSleep(aeEventLoop*)+0x165) [0x5561e3147335]
keydb-server 0.0.0.0:6379(aeProcessEvents+0x10b) [0x5561e3141b0b]
keydb-server 0.0.0.0:6379(aeMain+0x47) [0x5561e31484e7]
keydb-server 0.0.0.0:6379(workerThreadMain(void*)+0x73) [0x5561e32165b3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f046da0d6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f046d73661f]

------ INFO OUTPUT ------
# Server
redis_version:6.3.2
redis_git_sha1:d1dff8c2
redis_git_dirty:1
redis_build_id:60d46d3de2ba6e96
redis_mode:standalone
os:Linux 5.10.0-20-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:1
process_supervised:no
run_id:3d63993a80a3062526b120bbef752e3065bf5fba
tcp_port:6379
server_time_usec:1678789160011088
uptime_in_seconds:56
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:1067559
executable:/data/keydb-server
config_file:/etc/keydb/redis.conf

# Clients
connected_clients:8
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:56
client_recent_max_output_buffer:0
blocked_clients:5
tracking_clients:0
clients_in_timeout_table:5
current_client_thread:0
thread_0_clients:8
thread_1_clients:0

# Memory
used_memory:2447184
used_memory_human:2.33M
used_memory_rss:24010752
used_memory_rss_human:22.90M
used_memory_peak:2534360
used_memory_peak_human:2.42M
used_memory_peak_perc:96.56%
used_memory_overhead:2376138
used_memory_startup:2199232
used_memory_dataset:71046
used_memory_dataset_perc:28.65%
allocator_allocated:2858136
allocator_active:3395584
allocator_resident:7131136
total_system_memory:16785612800
total_system_memory_human:15.63G
used_memory_lua:43008
used_memory_lua_human:42.00K
used_memory_scripts:360
used_memory_scripts_human:360B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.19
allocator_frag_bytes:537448
allocator_rss_ratio:2.10
allocator_rss_bytes:3735552
rss_overhead_ratio:3.37
rss_overhead_bytes:16879616
mem_fragmentation_ratio:10.13
mem_fragmentation_bytes:21640728
mem_not_counted_for_evict:1048576
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:164186
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:1
rdb_bgsave_in_progress:0
rdb_last_save_time:1678789104
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:23
total_commands_processed:91
instantaneous_ops_per_sec:0
total_net_input_bytes:3996
total_net_output_bytes:484
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:9
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:104
total_writes_processed:84
instantaneous_lock_contention:2
avg_lock_contention:0.265625
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:778b111632ae9c2eeee64be3b358aadcde4b44db
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.150568
used_cpu_user:0.230824
used_cpu_sys_children:0.002011
used_cpu_user_children:0.000000
server_threads:2
long_lock_waits:0
used_cpu_sys_main_thread:0.065866
used_cpu_user_main_thread:0.131732

# Modules

# Commandstats
cmdstat_get:calls=9,usec=65,usec_per_call=7.22,rejected_calls=0,failed_calls=0
cmdstat_set:calls=1,usec=50,usec_per_call=50.00,rejected_calls=0,failed_calls=0
cmdstat_brpop:calls=44,usec=538,usec_per_call=12.23,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=1,usec=259,usec_per_call=259.00,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=11,usec=15,usec_per_call=1.36,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=1,usec=141,usec_per_call=141.00,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=1,usec=75,usec_per_call=75.00,rejected_calls=0,failed_calls=1
cmdstat_auth:calls=23,usec=174,usec_per_call=7.57,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_NOSCRIPT:count=1

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=1,avg_ttl=19994,cached_keys=1

# KeyDB
mvcc_depth:1

------ CLIENT LIST OUTPUT ------
id=13 addr=10.233.69.27:40700 laddr=10.233.67.110:6379 fd=24 name= age=30 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=19 obl=0 oll=0 omem=0 tot-mem=61475 events=r cmd=keys user=default redir=-1
id=14 addr=10.233.69.27:40710 laddr=10.233.67.110:6379 fd=25 name= age=30 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=brpop user=default redir=-1
id=10 addr=10.233.69.27:40692 laddr=10.233.67.110:6379 fd=21 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=21 obl=0 oll=0 omem=0 tot-mem=20533 events=r cmd=brpop user=default redir=-1
id=11 addr=10.233.67.28:41004 laddr=10.233.67.110:6379 fd=22 name= age=30 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=brpop user=default redir=-1
id=12 addr=10.233.67.28:41012 laddr=10.233.67.110:6379 fd=23 name= age=30 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=brpop user=default redir=-1
id=7 addr=10.233.67.28:40980 laddr=10.233.67.110:6379 fd=18 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=21 obl=0 oll=0 omem=0 tot-mem=20533 events=r cmd=brpop user=default redir=-1
id=8 addr=10.233.67.28:40994 laddr=10.233.67.110:6379 fd=19 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=24 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
id=9 addr=10.233.69.27:40690 laddr=10.233.67.110:6379 fd=20 name= age=33 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=24 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
1:16:M 14 Mar 2023 10:19:20.014 # main thread terminated
1:16:M 14 Mar 2023 10:19:20.014 # Bio thread for job type #0 terminated
1:16:M 14 Mar 2023 10:19:20.014 # Bio thread for job type #1 terminated
1:16:M 14 Mar 2023 10:19:20.014 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== KEYDB BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/JohnSully/KeyDB/issues

  Suspect RAM error? Use keydb-server --test-memory to verify it.

Aditional information

chart keydb-0.46.1

app version 6.3.2

additional values:

fullnameOverride: pplt-keydb
nodes: 1
password: AA000000aaaaaaa
podDisruptionBudget:
  enabled: false

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions