Skip to content

[CRASH] Redis-Server Crashed after failed to save rdb when receive shutdown command #10439

@warriorguo

Description

@warriorguo

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
12396:M 18 Mar 2022 12:24:56.236 # Redis 255.255.255 crashed by signal: 11, si_code: 1
12396:M 18 Mar 2022 12:24:56.236 # Accessing address: 0x60
12396:M 18 Mar 2022 12:24:56.236 # Killed by PID: 0, UID: 0
12396:M 18 Mar 2022 12:24:56.236 # Crashed running the instruction at: 0x1092fae2a

------ STACK TRACE ------
EIP:
0   redis-server                        0x00000001092fae2a call + 650

Backtrace:
0   libsystem_platform.dylib            0x00007ff81b6b0e2d _sigtramp + 29
1   ???                                 0x692065726f6d2072 0x0 + 7575166115407274098
2   redis-server                        0x00000001092fbd3f processCommand + 2735
3   redis-server                        0x0000000109316c5a processInputBuffer + 362
4   redis-server                        0x000000010930e853 readQueryFromClient + 1123
5   redis-server                        0x00000001093c8a86 connSocketEventHandler + 294
6   redis-server                        0x00000001092f0a24 aeProcessEvents + 772
7   redis-server                        0x00000001092f0c9d aeMain + 29
8   redis-server                        0x0000000109302b23 main + 2083
9   dyld                                0x0000000117c1f4fe start + 462

------ REGISTERS ------
12396:M 18 Mar 2022 12:24:56.244 #
RAX:0000000000000000 RBX:000000010948b3a8
RCX:0000000000000098 RDX:0000000000000000
RDI:0000600000741440 RSI:0000000000000000
RBP:00007ff7b6c14600 RSP:00007ff7b6c145a0
R8 :00000da5fc4a7acc R9 :000000000086fbf9
R10:00000000000007fb R11:0000000000000400
R12:000000000000000f R13:00007fad097045e0
R14:000000010949e690 R15:0000000000000098
RIP:00000001092fae2a EFL:0000000000010283
CS :000000000000002b FS:0000000000000000  GS:0000000000000000
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145af) -> 00000001093168d8
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145ae) -> 0000000000000001
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145ad) -> 00000001092fbd3f
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145ac) -> 00007ff7b6c14670
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145ab) -> 00000000ffffffff
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145aa) -> 00007fad097045e0
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a9) -> 0000000000000000
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a8) -> 000000010949e690
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a7) -> 0000000000000000
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a6) -> 0000000000950013
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a5) -> 0000000000000014
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a4) -> 0000000000000014
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a3) -> 000000010948b3a8
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a2) -> 0000000000000000
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a1) -> 0000600000039a1a
12396:M 18 Mar 2022 12:24:56.244 # (00007ff7b6c145a0) -> 0000000062340998

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:b69636d3
redis_git_dirty:1
redis_build_id:b32c284c00f7e639
redis_mode:standalone
os:Darwin 21.2.0 x86_64
arch_bits:64
multiplexing_api:kqueue
atomicvar_api:c11-builtin
gcc_version:4.2.1
process_id:12396
process_supervised:no
run_id:1459808a1bfb61d6893d39363cddc30bdf24f659
tcp_port:21111
server_time_usec:1647577496236058
uptime_in_seconds:0
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3410328
executable:/***/backend/redis/src/redis-server
config_file:/***/backend/redis/./tests/tmp/redis.conf.12377.2
io_threads_active:0

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:16896
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:1233776
used_memory_human:1.18M
used_memory_rss:3239936
used_memory_rss_human:3.09M
used_memory_peak:1233776
used_memory_peak_human:1.18M
used_memory_peak_perc:100.16%
used_memory_overhead:1106744
used_memory_startup:1071472
used_memory_dataset:127032
used_memory_dataset_perc:78.27%
allocator_allocated:1231632
allocator_active:3202048
allocator_resident:3202048
total_system_memory:8589934592
total_system_memory_human:8.00G
used_memory_lua:37888
used_memory_vm_eval:37888
used_memory_lua_human:37.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:37888
used_memory_vm_total:75776
used_memory_vm_total_human:74.00K
used_memory_functions:216
used_memory_scripts:216
used_memory_scripts_human:216B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:2.60
allocator_frag_bytes:1970416
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.01
rss_overhead_bytes:37888
mem_fragmentation_ratio:2.63
mem_fragmentation_bytes:2008304
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:34000
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:20
rdb_bgsave_in_progress:0
rdb_last_save_time:1647577496
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:2
total_commands_processed:24
instantaneous_ops_per_sec:14
total_net_input_bytes:656
total_net_output_bytes:4856
instantaneous_input_kbps:0.38
instantaneous_output_kbps:2.93
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:26
total_writes_processed:24
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:0
reply_buffer_expands:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:d12e064dd81600a754c62ac66347eedf9ccb70b8
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.006237
used_cpu_user:0.005119
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Modules

# Commandstats
cmdstat_select:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=1,usec=49,usec_per_call=49.00,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=1,usec=3,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_shutdown:calls=0,usec=0,usec_per_call=0.00,rejected_calls=0,failed_calls=1
cmdstat_info:calls=1,usec=68,usec_per_call=68.00,rejected_calls=0,failed_calls=0
cmdstat_set:calls=20,usec=63,usec_per_call=3.15,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_ERR:count=1

# Latencystats
latency_percentiles_usec_select:p50=6.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_config|get:p50=49.151,p99=49.151,p99.9=49.151
latency_percentiles_usec_ping:p50=3.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_info:p50=68.095,p99=68.095,p99.9=68.095
latency_percentiles_usec_set:p50=3.007,p99=10.047,p99.9=10.047

# Cluster
cluster_enabled:0

# Keyspace
db9:keys=20,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=4 addr=127.0.0.1:52730 laddr=127.0.0.1:21111 fd=12 name= age=0 idle=0 flags=u db=9 sub=0 psub=0 multi=-1 qbuf=18 qbuf-free=16872 argv-mem=0 multi-mem=0 rbs=16384 rbp=45 obl=45 oll=0 omem=0 tot-mem=34000 events=r cmd=shutdown user=default redir=-1 resp=2

------ CURRENT CLIENT INFO ------
id=4 addr=127.0.0.1:52730 laddr=127.0.0.1:21111 fd=12 name= age=0 idle=0 flags=u db=9 sub=0 psub=0 multi=-1 qbuf=18 qbuf-free=16872 argv-mem=0 multi-mem=0 rbs=16384 rbp=45 obl=45 oll=0 omem=0 tot-mem=34000 events=r cmd=shutdown user=default redir=-1 resp=2

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-user-flush no
io-threads 1
lazyfree-lazy-user-del no
io-threads-do-reads no
lazyfree-lazy-expire no
activedefrag no
lazyfree-lazy-eviction no
slave-read-only yes
repl-diskless-sync yes
list-compress-depth 0
sanitize-dump-payload no
client-query-buffer-limit 1gb
lazyfree-lazy-server-del no
proto-max-bulk-len 512mb
repl-diskless-load disabled
replica-read-only yes

------ DUMPING CODE AROUND EIP ------
Symbol: call (base: 0x1092faba0)
Module: /***/redis/src/redis-server (base 0x1092eb000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x1092faba0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
12396:M 18 Mar 2022 12:24:56.245 # dump of function (hexdump of 778 bytes):
554889e54157415641554154534883ec384189f74989fd488b9780000000488b87e0000000488945b04825ff3fe7ff488987e00000004c8d35b33a1a00418b8ee800000089f083e010894dd409c8750b41c786e800000001000000488955b8498b86380b0000488945c0498b8698070000488905f8d51900498b8628040000488d480149898e280400004885c0756b488d7da031f6e87e261100486945a040420f0048634da84801c148bacff753e3a59bc4204889c848f7ea49898ee00e00004889d048c1e83f48c1fa074801c2498996d80e000048badb34b6d782de1b434889c848f7ea4889d048c1e83f48c1fa124801c2498996c00e000041838630040000014c8d25bf391a0041ff14244889c3498b45704c89efff505041ff14244829d84889c2498985c8000000498b86380b0000488945c841838630040000ff498b8698070000488b5db84885db741a483b051bd519007e1148838380010000014889050ad51900eb1948890501d519004983bdb00000000074084883838001000001498b85e0000000480fbae028731848b9bffffffffffeffff4821c84883c840498985e00000004589fc4183e4fca900010000450f44e74183bea405000000450f44e7a9000100004989d77423498b8e900f00004885c974176685c00f8839030000a90040000074078089e10000004041f6c4010f848c000000498b8ec00f000048bacff753e3a59bc4204c89f848f7ea4885c974344889d048c1e83f48c1fa074801c24839ca7c21f7436000400000488d058a5a1100488d3d765a1100480f44f84889d6e8c65c090041f685e0000000107532f6436110752c498b45604889c64885c07504498b7550498d4d5c498d55484885c0480f44ca8b114c89ef4c89f9e81a8b0700498b45700fb74060a910080000741541f685e000000010744d41f6c4027559e9e3000000498b45604889c14885c07504498b4d50498d555c498d75484885c0480f44d6448b02498bb6e0020000498b45188b50284c89efe86ea9020041f685e00000001075b34c89efe8ac76010041f6c4020f848f000000f30f6f8368010000660f6f0d
Function at 0x109390ab0 is latencyAddSample
Function at 0x109373940 is slowlogPushEntryIfNeeded
Function at 0x1093257f0 is replicationFeedMonitors
Function at 0x109312540 is freeClientOriginalArgv

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Additional information

  1. OS distribution and version
    Mac Monterey 12.1, Redis version: Redis 7.0rc2
  2. Steps to reproduce
    a. launch the Redis process with nobody, sudo -u nobody src/redis-server
    b. set the rdb file to readonly, for example chmod 0444 dump.rdb
    c. send shutdown command to the server, the server will crash directly

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions