Ceph » RADOS

In the https://tracker.ceph.com/issues/55407#note-14 an OSD crash during early boot up is reported:

...
   -79> 2022-05-16T09:10:21.707+0000 7fb1f1728640 10 monclient: _renew_subs
   -78> 2022-05-16T09:10:21.707+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -77> 2022-05-16T09:10:21.707+0000 7fb1f5730640  3 osd.3 1739257 handle_osd_map epochs [1739258,1739297], i have 1739257, src has [1491297,1739460]
   -76> 2022-05-16T09:10:21.751+0000 7fb1f1728640 10 monclient: _renew_subs
   -75> 2022-05-16T09:10:21.751+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -74> 2022-05-16T09:10:21.751+0000 7fb1f5730640  3 osd.3 1739297 handle_osd_map epochs [1739298,1739337], i have 1739297, src has [1491297,1739460]
   -73> 2022-05-16T09:10:21.807+0000 7fb1f1728640 10 monclient: _renew_subs
   -72> 2022-05-16T09:10:21.807+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -71> 2022-05-16T09:10:21.811+0000 7fb1f5730640  3 osd.3 1739337 handle_osd_map epochs [1739338,1739377], i have 1739337, src has [1491297,1739460]
   -70> 2022-05-16T09:10:21.859+0000 7fb1f1728640 10 monclient: _renew_subs
   -69> 2022-05-16T09:10:21.859+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -68> 2022-05-16T09:10:21.859+0000 7fb1f5730640  3 osd.3 1739377 handle_osd_map epochs [1739378,1739417], i have 1739377, src has [1491297,1739460]
   -67> 2022-05-16T09:10:21.903+0000 7fb1f1728640 10 monclient: _renew_subs
   -66> 2022-05-16T09:10:21.903+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -65> 2022-05-16T09:10:21.915+0000 7fb1f5730640  3 osd.3 1739417 handle_osd_map epochs [1739418,1739457], i have 1739417, src has [1491297,1739460]
   -64> 2022-05-16T09:10:21.967+0000 7fb1f1728640 10 monclient: _renew_subs
   -63> 2022-05-16T09:10:21.967+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -62> 2022-05-16T09:10:21.967+0000 7fb1f5730640  3 osd.3 1739457 handle_osd_map epochs [1739458,1739460], i have 1739457, src has [1491297,1739460]
   -61> 2022-05-16T09:10:21.991+0000 7fb1f1728640  5 osd.3 1739460 heartbeat osd_stat(store_statfs(0xe7a54d6000/0x0/0xe8e0db6000, data 0xab32b8c9/0xf770e000, compress 0x0/0x0/0x0, omap 0x0, meta 0x441d0000), peers [] op hist [])
   -60> 2022-05-16T09:10:21.991+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -59> 2022-05-16T09:10:21.991+0000 7fb1f1728640 10 monclient: _renew_subs
   -58> 2022-05-16T09:10:21.991+0000 7fb1f1728640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -57> 2022-05-16T09:10:22.027+0000 7fb1f5730640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -56> 2022-05-16T09:10:22.031+0000 7fb1f5730640  1 osd.3 1739460 start_boot
   -55> 2022-05-16T09:10:22.031+0000 7fb1f5730640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -54> 2022-05-16T09:10:22.031+0000 7fb1f5730640 10 monclient: handle_get_version_reply finishing 2 version 1739460
   -53> 2022-05-16T09:10:22.031+0000 7fb201194640  5 osd.3 1739460 heartbeat osd_stat(store_statfs(0xe7a54d6000/0x0/0xe8e0db6000, data 0xab32b8c9/0xf770e000, compress 0x0/0x0/0x0, omap 0x0, meta 0x441d0000), peers [] op hist [])
   -52> 2022-05-16T09:10:22.039+0000 7fb1f472e640 10 monclient: tick
   -51> 2022-05-16T09:10:22.039+0000 7fb1f472e640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-05-16T09:09:52.041726+0000)
   -50> 2022-05-16T09:10:22.039+0000 7fb1f9f39640 -1 osd.3 1739460 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
   -49> 2022-05-16T09:10:22.039+0000 7fb1f9f39640  1 osd.3 1739460 set_numa_affinity setting numa affinity to node 0 cpus 0-19
   -48> 2022-05-16T09:10:22.063+0000 7fb1f9f39640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -47> 2022-05-16T09:10:22.411+0000 7fb1f5f31640  5 prioritycache tune_memory target: 2147483648 mapped: 378167296 unmapped: 335536128 heap: 713703424 old mem: 1020054731 new mem: 1020054731
   -46> 2022-05-16T09:10:22.499+0000 7fb1f3f2d640  4 mgrc reconnect Starting new session with [v2:172.16.99.10:6842/11647,v1:172.16.99.10:6843/11647]
   -45> 2022-05-16T09:10:22.499+0000 7fb202196640 10 monclient: get_auth_request con 0x561b0af07c00 auth_method 0
   -44> 2022-05-16T09:10:22.503+0000 7fb1f5730640  4 mgrc ms_handle_reset ms_handle_reset con 0x561b0af07c00
   -43> 2022-05-16T09:10:22.503+0000 7fb1f5730640  4 mgrc reconnect Terminating session with v2:172.16.99.10:6842/11647
   -42> 2022-05-16T09:10:22.503+0000 7fb1f5730640  4 mgrc reconnect waiting to retry connect until 2713.683826s
   -41> 2022-05-16T09:10:22.847+0000 7fb1fe742640  1 osd.3 1739460 tick checking mon for new map
   -40> 2022-05-16T09:10:23.039+0000 7fb1f472e640 10 monclient: tick
   -39> 2022-05-16T09:10:23.039+0000 7fb1f472e640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-05-16T09:09:53.041842+0000)
   -38> 2022-05-16T09:10:23.059+0000 7fb1f5f31640  5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.4
   -37> 2022-05-16T09:10:23.059+0000 7fb1f5f31640  5 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.168421
   -36> 2022-05-16T09:10:23.059+0000 7fb1f5f31640  5 bluestore.MempoolThread(0x561b09d1cb38) _resize_shards cache_size: 1020054731 kv_alloc: 398458880 kv_used: 24928 kv_onode_alloc: 167772160 kv_onode_used: 8276112 meta_alloc: 339738624 meta_used: 13100328 data_alloc: 104857600 data_used: 0
   -35> 2022-05-16T09:10:23.099+0000 7fb202196640 10 monclient: handle_auth_request added challenge on 0x561b0e47cc00
   -34> 2022-05-16T09:10:23.099+0000 7fb203198640 10 monclient: handle_auth_request added challenge on 0x561b0e47d000
   -33> 2022-05-16T09:10:23.131+0000 7fb1f5730640 10 monclient: _renew_subs
   -32> 2022-05-16T09:10:23.131+0000 7fb1f5730640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -31> 2022-05-16T09:10:23.131+0000 7fb1f5730640  3 osd.3 1739460 handle_osd_map epochs [1739461,1739461], i have 1739460, src has [1491297,1739461]
   -30> 2022-05-16T09:10:23.151+0000 7fb1f1728640  1 osd.3 1739461 state: booting -> active
   -29> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1d576400 auth_method 0
   -28> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b298b0c00 auth_method 0
   -27> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1d577c00 auth_method 0
   -26> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1a61a000 auth_method 0
   -25> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1a61b800 auth_method 0
   -24> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1a61ac00 auth_method 0
   -23> 2022-05-16T09:10:23.151+0000 7fb202997640 10 monclient: get_auth_request con 0x561b1d577000 auth_method 0
   -22> 2022-05-16T09:10:23.187+0000 7fb1f5730640  3 osd.3 1739461 handle_osd_map epochs [1739461,1739461], i have 1739461, src has [1491297,1739461]
   -21> 2022-05-16T09:10:23.411+0000 7fb1f5f31640  5 prioritycache tune_memory target: 2147483648 mapped: 378462208 unmapped: 335241216 heap: 713703424 old mem: 1020054731 new mem: 1020054731
   -20> 2022-05-16T09:10:23.499+0000 7fb1f3f2d640  4 mgrc reconnect Starting new session with [v2:172.16.99.10:6842/11647,v1:172.16.99.10:6843/11647]
   -19> 2022-05-16T09:10:23.547+0000 7fb1fb73c640 10 monclient: _send_mon_message to mon.blue-compute at v2:172.16.0.119:3300/0
   -18> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1d576800 auth_method 0
   -17> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1d577800 auth_method 0
   -16> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1d577400 auth_method 0
   -15> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1a61a800 auth_method 0
   -14> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b218fe000 auth_method 0
   -13> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b0e47d800 auth_method 0
   -12> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b0e47dc00 auth_method 0
   -11> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1a61a400 auth_method 0
   -10> 2022-05-16T09:10:23.595+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1a61bc00 auth_method 0
    -9> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1a61b400 auth_method 0
    -8> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1d576c00 auth_method 0
    -7> 2022-05-16T09:10:23.595+0000 7fb203198640 10 monclient: get_auth_request con 0x561b1d576000 auth_method 0
    -6> 2022-05-16T09:10:23.599+0000 7fb203198640 10 monclient: get_auth_request con 0x561b0af07c00 auth_method 0
    -5> 2022-05-16T09:10:23.599+0000 7fb202196640 10 monclient: get_auth_request con 0x561b1a61b000 auth_method 0
    -4> 2022-05-16T09:10:23.599+0000 7fb1f5730640  4 mgrc handle_mgr_configure stats_period=5
    -3> 2022-05-16T09:10:23.699+0000 7fb203198640 10 monclient: handle_auth_request added challenge on 0x561b1fe40400
    -2> 2022-05-16T09:10:23.699+0000 7fb202997640 10 monclient: handle_auth_request added challenge on 0x561b1fe40c00
    -1> 2022-05-16T09:10:23.699+0000 7fb202196640 10 monclient: handle_auth_request added challenge on 0x561b1fe40800
     0> 2022-05-16T09:10:23.711+0000 7fb202997640 -1 *** Caught signal (Segmentation fault) **
 in thread 7fb202997640 thread_name:msgr-worker-1

 ceph version 17.0.0-12154-g6f78f2f42ea (6f78f2f42eaa5f84d25bf175563df9717f8c4203) quincy (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb2044a8520]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_pwl
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 immutable_obj_cache
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/ 5 rgw_datacache
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 fuse
   2/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   1/ 5 prioritycache
   0/ 5 test
   0/ 5 cephfs_mirror
   0/ 5 cephsqlite
   0/ 5 seastore
   0/ 5 seastore_onode
   0/ 5 seastore_odata
   0/ 5 seastore_omap
   0/ 5 seastore_tm
   0/ 5 seastore_t
   0/ 5 seastore_cleaner
   0/ 5 seastore_lba
   0/ 5 seastore_lba_details
   0/ 5 seastore_cache
   0/ 5 seastore_journal
   0/ 5 seastore_device
   0/ 5 seastore_backref
   0/ 5 alienstore
   1/ 5 mclock
   0/ 5 cyanstore
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  7fb1e5f11640 / osd_srv_heartbt
  7fb1f1728640 / cfin
  7fb1f1f29640 / bstore_kv_sync
  7fb1f3f2d640 / safe_timer
  7fb1f472e640 / safe_timer
  7fb1f5730640 / ms_dispatch
  7fb1f5f31640 / bstore_mempool
  7fb1f6732640 / rocksdb:high0
  7fb1f6f33640 / rocksdb:low1
  7fb1f7734640 / rocksdb:low0
  7fb1f9f39640 / fn_anonymous
  7fb1fb73c640 / safe_timer
  7fb1fe742640 / safe_timer
  7fb201194640 / io_context_pool
  7fb202196640 / msgr-worker-2
  7fb202997640 / msgr-worker-1
  7fb203198640 / msgr-worker-0
  max_recent     10000
  max_new        10000
  log_file /var/lib/ceph/crash/2022-05-16T09:10:23.706211Z_777e42fd-25c5-4efc-a27e-b02c6eb3110a/log
--- end dump of recent events ---
Violación de segmento (`core' generado)

This happens on master:

Merge: 12d0955b5fc 89430b1c39d
Author: Samuel Just <sjust@redhat.com>
Date:   Wed May 11 19:06:53 2022 -0700

The invocation was:

root@red-compute:/mnt/ceph-recovery# LD_LIBRARY_PATH=lib/ bin/ceph-osd -d --cluster ceph --id 3 --default-log-to-stderr=true --err-to-stderr=true --default-log-to-file=false --foreground --setuser ceph --setgroup ceph --version
ceph version 17.0.0-12154-g6f78f2f42ea (6f78f2f42eaa5f84d25bf175563df9717f8c4203) quincy (dev)
root@red-compute:/mnt/ceph-recovery# ls