Bug #52221: crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end()) - RADOS - Ceph

Actions

Copy link

Bug #52221

open

crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end())

Added by Telemetry Bot over 4 years ago. Updated almost 3 years ago.

Status:

Triaged

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Telemetry

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v13.2.10, Ceph - v13.2.6, Ceph - v13.2.8, Ceph - v14.2.1, Ceph - v14.2.10, Ceph - v14.2.11, Ceph - v14.2.15, Ceph - v14.2.16, Ceph - v14.2.18, Ceph - v14.2.19, Ceph - v14.2.2, Ceph - v14.2.22, Ceph - v14.2.3, Ceph - v14.2.4, Ceph - v14.2.5, Ceph - v14.2.7, Ceph - v14.2.8, Ceph - v14.2.9, Ceph - v15.1.0, Ceph - v15.2.0, Ceph - v15.2.1, Ceph - v15.2.10, Ceph - v15.2.11, Ceph - v15.2.12, Ceph - v15.2.13, Ceph - v15.2.14, Ceph - v15.2.15, Ceph - v15.2.16, Ceph - v15.2.17, Ceph - v15.2.2, Ceph - v15.2.3, Ceph - v15.2.4, Ceph - v15.2.5, Ceph - v15.2.6, Ceph - v15.2.7, Ceph - v15.2.8, Ceph - v15.2.9, Ceph - v16.2.0, Ceph - v16.2.1, Ceph - v16.2.10, Ceph - v16.2.11, Ceph - v16.2.12, Ceph - v16.2.3, Ceph - v16.2.4, Ceph - v16.2.5, Ceph - v16.2.6, Ceph - v16.2.7, Ceph - v16.2.9, Ceph - v17.0.0, Ceph - v17.2.0, Ceph - v17.2.1, Ceph - v17.2.2, Ceph - v17.2.3, Ceph - v17.2.4, Ceph - v17.2.5, Ceph - v17.2.6

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Tags (freeform):

Merge Commit:

Fixed In:

Released In:

Upkeep Timestamp:

Tags:

test-failure

Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=9ee7dc6ce5b80b3a4a423d804952ac1ea7bcdbf73c236084bd1169faaf9238c8

Assert condition: p != added_maps_bl.end()
Assert function: void OSD::handle_osd_map(MOSDMap*)

Sanitized backtrace:

    /lib/x86_64-linux-gnu/libpthread.so.0(
    /usr/bin/ceph-osd(
    OSD::handle_osd_map(MOSDMap*)
    OSD::ms_dispatch(Message*)
    Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)
    DispatchQueue::entry()
    DispatchQueue::DispatchThread::entry()
    /lib/x86_64-linux-gnu/libpthread.so.0(
    clone()

Crash dump sample:

{
    "assert_condition": "p != added_maps_bl.end()",
    "assert_file": "osd/OSD.cc",
    "assert_func": "void OSD::handle_osd_map(MOSDMap*)",
    "assert_line": 8050,
    "assert_msg": "osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 7faacbb1a640 time 2021-08-05T22:32:23.812100+0000\nosd/OSD.cc: 8050: FAILED ceph_assert(p != added_maps_bl.end())",
    "assert_thread_name": "ms_dispatch",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x141f0) [0x7faad96e71f0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x172) [0x561f889998cc]",
        "/usr/bin/ceph-osd(+0x516a2b) [0x561f88999a2b]",
        "(OSD::handle_osd_map(MOSDMap*)+0x13c1) [0x561f88ac6dc1]",
        "(OSD::ms_dispatch(Message*)+0x80) [0x561f88adb840]",
        "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc3) [0x561f894a8963]",
        "(DispatchQueue::entry()+0x5ef) [0x561f894a72cf]",
        "(DispatchQueue::DispatchThread::entry()+0x11) [0x561f892d7e41]",
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x9450) [0x7faad96dc450]",
        "clone()" 
    ],
    "ceph_version": "16.2.4",
    "crash_id": "2021-08-05T22:32:23.820514Z_1051405e-5319-49e5-8438-89a7c5ae644f",
    "entity_name": "osd.8a780e602eed4f0e012a3c9cd38ff6fabdb84c97",
    "os_id": "ubuntu",
    "os_name": "Ubuntu",
    "os_version": "21.04 (Hirsute Hippo)",
    "os_version_id": "21.04",
    "process_name": "ceph-osd",
    "stack_sig": "3d620390bfce4aec3e0cbc531204df0feaf8a4eeb60f66bf4a2b9890bf89fe92",
    "timestamp": "2021-08-05T22:32:23.820514Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.11.0-25-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#27-Ubuntu SMP Fri Jul 9 23:06:29 UTC 2021" 
}

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Telemetry Bot over 4 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v16.2.1, v16.2.4, v16.2.5 added

Actions

Copy link

Updated by Neha Ojha over 4 years ago

Has duplicate Bug #52143: crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end()) added

Actions

Copy link

Updated by Josh Durgin over 4 years ago

Status changed from New to Triaged

Actions

Copy link

Updated by Neha Ojha over 4 years ago

Has duplicate Bug #52200: crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end()) added

Actions

Copy link

Updated by Neha Ojha over 4 years ago

Has duplicate Bug #52186: crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end()) added

Actions

Copy link

Updated by Telemetry Bot about 4 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v13.2.10, v13.2.6, v13.2.8, v14.2.1, v14.2.10, v14.2.11, v14.2.15, v14.2.16, v14.2.18, v14.2.2, v14.2.22, v14.2.3, v14.2.4, v14.2.5, v14.2.7, v14.2.9, v15.1.0, v15.2.10, v15.2.11, v15.2.13, v15.2.14, v15.2.15, v15.2.2, v15.2.3, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8, v15.2.9 added

Actions

Copy link

Updated by Telemetry Bot about 4 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v16.2.0, v16.2.6, v16.2.7 added

Actions

Copy link

Updated by Telemetry Bot about 4 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v17.0.0 added

Actions

Copy link

Updated by Telemetry Bot about 4 years ago

Crash signature (v1) updated (diff)

Actions

Copy link

#10

Updated by Telemetry Bot about 4 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)

Actions

Copy link

#11

Updated by Telemetry Bot about 4 years ago

Crash signature (v1) updated (diff)
Affected Versions v14.2.19, v14.2.8, v15.2.0, v15.2.1, v15.2.12 added

Actions

Copy link

#12

Updated by Telemetry Bot over 3 years ago

Crash signature (v1) updated (diff)
Affected Versions v15.2.16, v16.2.3, v16.2.9, v17.2.0 added

Actions

Copy link

#13

Updated by Telemetry Bot over 3 years ago

Crash signature (v1) updated (diff)
Affected Versions v17.2.1 added

Actions

Copy link

#14

Updated by Laura Flores about 3 years ago

Tags set to test-failure
Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)

/a/yuriw-2023-01-27_16:33:50-rados-wip-yuri2-testing-2023-01-26-1532-distro-default-smithi/7142354

4/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2064-g161d7183/rpm/el8/BUILD/ceph-18.0.0-2064-g161d7183/src/osd/OSD.cc: 7972: FAILED ceph_assert(p != added_maps_bl.end())
2023-01-28T12:39:37.178 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:
2023-01-28T12:39:37.178 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  ceph version 18.0.0-2064-g161d7183 (161d7183ad670eccbfb8e194b7688d213eb4d366) reef (dev)
2023-01-28T12:39:37.178 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x55851be0d2ff]
2023-01-28T12:39:37.179 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  2: /usr/bin/ceph-osd(+0x5da4c5) [0x55851be0d4c5]
2023-01-28T12:39:37.179 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  3: (OSD::handle_osd_map(MOSDMap*)+0x3230) [0x55851bf478c0]
2023-01-28T12:39:37.179 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  4: (OSD::ms_dispatch(Message*)+0x78) [0x55851bf48ff8]
2023-01-28T12:39:37.179 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  5: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc9) [0x55851ca3b129]
2023-01-28T12:39:37.179 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  6: (DispatchQueue::entry()+0x50f) [0x55851ca398df]
2023-01-28T12:39:37.180 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  7: (DispatchQueue::DispatchThread::entry()+0x11) [0x55851c84c811]
2023-01-28T12:39:37.180 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  8: /lib64/libpthread.so.0(+0x81ca) [0x7f7775bce1ca]
2023-01-28T12:39:37.180 INFO:journalctl@ceph.osd.1.smithi100.stdout:Jan 28 12:39:36 smithi100 ceph-3bc1d294-9f08-11ed-9e56-001a4aab830c-osd-1[122992]:  9: clone()

From osd.1's log:

   -23> 2023-01-28T12:39:36.583+0000 7f7767697700 10 osd.1 23 tick_without_osd_lock
   -22> 2023-01-28T12:39:36.583+0000 7f7767697700 20 osd.1 23 scrub_random_backoff lost coin flip, randomly backing off (ratio: 0.66)
   -21> 2023-01-28T12:39:36.583+0000 7f7767697700 10 osd.1 23 promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0 B; target 25 obj/sec or 5 MiB/sec
   -20> 2023-01-28T12:39:36.583+0000 7f7767697700 20 osd.1 23 promote_throttle_recalibrate  new_prob 1000
   -19> 2023-01-28T12:39:36.583+0000 7f7767697700 10 osd.1 23 promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted new_prob 1000, prob 1000 -> 1000
   -18> 2023-01-28T12:39:36.618+0000 7f776268d700  5 prioritycache tune_memory target: 4294967296 mapped: 33767424 unmapped: 1146880 heap: 34914304 old mem: 2845415832 new mem: 2845415832
   -17> 2023-01-28T12:39:36.785+0000 7f7767e98700 10 osd.1 23 tick
   -16> 2023-01-28T12:39:36.785+0000 7f7767e98700 20 osd.1 23 tick last_purged_snaps_scrub 2023-01-28T12:38:49.528649+0000 next 2023-01-29T21:14:12.966149+0000
   -15> 2023-01-28T12:39:36.885+0000 7f776e8a8700  1 -- [v2:172.21.15.100:6816/984051129,v1:172.21.15.100:6817/984051129] <== osd.0 172.21.15.100:0/4079546837 18 ==== osd_ping(ping e24 up_from 8 ping_stamp 2023-01-28T12:39:36.886795+0000/71.413482666s send_stamp 71.413482666s delta_ub 23.045795441s) v5 ==== 2033+0+0 (crc 0 0 0) 0x558520545080 con 0x55852054c400
   -14> 2023-01-28T12:39:36.885+0000 7f776f0a9700  1 -- [v2:172.21.15.100:6814/984051129,v1:172.21.15.100:6815/984051129] <== osd.0 172.21.15.100:0/4079546837 18 ==== osd_ping(ping e24 up_from 8 ping_stamp 2023-01-28T12:39:36.886795+0000/71.413482666s send_stamp 71.413482666s delta_ub 23.045795441s) v5 ==== 2033+0+0 (crc 0 0 0) 0x5585207dac00 con 0x55852054c800
   -13> 2023-01-28T12:39:36.885+0000 7f776e8a8700 20 osd.1 23 handle_osd_ping new stamps hbstamp(osd.0 up_from 8 peer_clock_delta [23.045286179s,23.045795441s])
   -12> 2023-01-28T12:39:36.885+0000 7f776e8a8700  1 -- [v2:172.21.15.100:6816/984051129,v1:172.21.15.100:6817/984051129] --> 172.21.15.100:0/4079546837 -- osd_ping(ping_reply e23 up_from 13 ping_stamp 2023-01-28T12:39:36.886795+0000/71.413482666s send_stamp 48.368198395s delta_ub -23.045286179s) v5 -- 0x558520545200 con 0x55852054c400
   -11> 2023-01-28T12:39:36.885+0000 7f776e8a8700 10 osd.1 23 maybe_share_map con 0x55852054d000 v2:172.21.15.100:6804/4079546837 map epoch 23 -> 24 (as per caller)
   -10> 2023-01-28T12:39:36.885+0000 7f776f0a9700 20 osd.1 23 handle_osd_ping new stamps hbstamp(osd.0 up_from 8 peer_clock_delta [23.045253754s,23.045795441s])
    -9> 2023-01-28T12:39:36.885+0000 7f776f0a9700  1 -- [v2:172.21.15.100:6814/984051129,v1:172.21.15.100:6815/984051129] --> 172.21.15.100:0/4079546837 -- osd_ping(ping_reply e23 up_from 13 ping_stamp 2023-01-28T12:39:36.886795+0000/71.413482666s send_stamp 48.368228912s delta_ub -23.045253754s) v5 -- 0x5585207da180 con 0x55852054c800
    -8> 2023-01-28T12:39:36.886+0000 7f775f7df700  1 -- [v2:172.21.15.100:6812/984051129,v1:172.21.15.100:6813/984051129] <== osd.0 v2:172.21.15.100:6804/4079546837 186 ==== osd_map(24..24 src has 1..24) v4 ==== 952+0+0 (crc 0 0 0) 0x558520f3b880 con 0x55852054d000
    -7> 2023-01-28T12:39:36.886+0000 7f775f7df700 20 osd.1 23 OSD::ms_dispatch: osd_map(24..24 src has 1..24) v4
    -6> 2023-01-28T12:39:36.886+0000 7f775f7df700 20 osd.1 23 _dispatch 0x558520f3b880 osd_map(24..24 src has 1..24) v4
    -5> 2023-01-28T12:39:36.886+0000 7f775f7df700  3 osd.1 23 handle_osd_map epochs [24,24], i have 23, src has [1,24]
    -4> 2023-01-28T12:39:36.886+0000 7f775f7df700 10 osd.1 23 handle_osd_map  got inc map for epoch 24
    -3> 2023-01-28T12:39:36.886+0000 7f7765693700 -1 bdev(0x55851fd8ae00 /var/lib/ceph/osd/ceph-1/block) _aio_thread got r=-5 ((5) Input/output error)
    -2> 2023-01-28T12:39:36.886+0000 7f7765693700 -1 bdev(0x55851fd8ae00 /var/lib/ceph/osd/ceph-1/block) _aio_thread translating the error to EIO for upper layer
    -1> 2023-01-28T12:39:36.888+0000 7f775f7df700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2064-g161d7183/rpm/el8/BUILD/ceph-18.0.0-2064-g161d7183/src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 7f775f7df700 time 2023-01-28T12:39:36.887483+0000
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2064-g161d7183/rpm/el8/BUILD/ceph-18.0.0-2064-g161d7183/src/osd/OSD.cc: 7972: FAILED ceph_assert(p != added_maps_bl.end())

 ceph version 18.0.0-2064-g161d7183 (161d7183ad670eccbfb8e194b7688d213eb4d366) reef (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x55851be0d2ff]
 2: /usr/bin/ceph-osd(+0x5da4c5) [0x55851be0d4c5]
 3: (OSD::handle_osd_map(MOSDMap*)+0x3230) [0x55851bf478c0]
 4: (OSD::ms_dispatch(Message*)+0x78) [0x55851bf48ff8]
 5: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc9) [0x55851ca3b129]
 6: (DispatchQueue::entry()+0x50f) [0x55851ca398df]
 7: (DispatchQueue::DispatchThread::entry()+0x11) [0x55851c84c811]
 8: /lib64/libpthread.so.0(+0x81ca) [0x7f7775bce1ca]
 9: clone()

Unfortunately, there was no coredump available.

Actions

Copy link

#15

Updated by Telemetry Bot almost 3 years ago

Crash signature (v1) updated (diff)
Affected Versions v15.2.17, v16.2.10, v16.2.11, v16.2.12, v17.2.3, v17.2.5 added

Actions

Copy link

#16

Updated by Telemetry Bot almost 3 years ago

Crash signature (v1) updated (diff)
Affected Versions v17.2.2, v17.2.4, v17.2.6 added

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » RADOS

Tags

Custom queries

Bug #52221

crash: void OSD::handle_osd_map(MOSDMap*): assert(p != added_maps_bl.end())

Updated by Telemetry Bot over 4 years ago

Updated by Neha Ojha over 4 years ago

Updated by Josh Durgin over 4 years ago

Updated by Neha Ojha over 4 years ago

Updated by Neha Ojha over 4 years ago

Updated by Telemetry Bot about 4 years ago

Updated by Telemetry Bot about 4 years ago

Updated by Telemetry Bot about 4 years ago

Updated by Telemetry Bot about 4 years ago

Updated by Telemetry Bot about 4 years ago

Updated by Telemetry Bot about 4 years ago

Updated by Telemetry Bot over 3 years ago

Updated by Telemetry Bot over 3 years ago

Updated by Laura Flores about 3 years ago

Updated by Telemetry Bot almost 3 years ago

Updated by Telemetry Bot almost 3 years ago