Ceph » RADOS

Greg Farnum wrote:

The monitor was out of quorum for 30 minutes; it probably has to do with holding on to client connections or else not cleaning up the session map from when it was last in. I'm not sure this is high priority though since it's a crash on shutdown in a failure scenario...

Unless you're looking at something different, the lost quorum happened during the test not at shutdown. The mds thrasher had just successfully thrashed (killed and standby took over successfully) an MDS around ~8 seconds earlier.

mon.c timeline:
2019-04-06 08:58:28.846 hits a lease timeout and triggers the election process
2019-04-06 08:58:28.846 first output of "probing" state
2019-04-06 08:58:28.850 first output of "electing" state
2019-04-06 09:27:34.942 crash output line

It does not output the "peon" or "leader" state again in those 29 minutes; it times out 291 elections and starts 294 during that time. I don't know why it happened but mon.c was out of quorum that whole time.

mon.c is failing to connect to mon.a:

2019-04-06 09:19:20.484 7f269f20d700  1 --2- [v2:172.21.15.180:3301/0,v1:172.21.15.180:6790/0] >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x3137200 0x2f77600 secure :-1 s=BANNER_CONNECTING pgs=3162 cs=280 l=0 rx=0x4171da0 tx=0x4b2b080)._handle_peer_banner_payload supported=0 required=0
2019-04-06 09:19:20.484 7f269f20d700  1 --2- [v2:172.21.15.180:3301/0,v1:172.21.15.180:6790/0] >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x3137200 0x2f77600 secure :-1 s=START_CONNECT pgs=3162 cs=281 l=0 rx=0x4171da0 tx=0x4b2b080)._fault waiting 15.000000

2019-04-06 09:19:00.906 7f8546b47700  1 --2- [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] >> [v2:172.21.15.180:3301/0,v1:172.21.15.180:6790/0] conn(0x361b680 0x33f9b80 secure :-1 s=BANNER_CONNECTING pgs=581 cs=284 l=0 rx=0x8ed3ab0 tx=0x4eb4b80)._handle_peer_banner_payload supported=0 required=0
2019-04-06 09:19:00.906 7f8546b47700  1 --2- [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] >> [v2:172.21.15.180:3301/0,v1:172.21.15.180:6790/0] conn(0x361b680 0x33f9b80 secure :-1 s=START_CONNECT pgs=581 cs=285 l=0 rx=0x8ed3ab0 tx=0x4eb4b80)._fault waiting 15.000000

(not surprisingly, MON_DOWN is in the ceph.log too, and the run would have failed with that had it not failed for some other reason. will keep an eye out for that!)

/ceph/teuthology-archive/pdonnell-2019-04-17_06:12:56-kcephfs-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3857629/teuthology.log

/a/yuriw-2019-06-07_19:41:42-rados-wip-yuri4-testing-2019-06-07-1600-nautilus-distro-basic-smithi/4012630/

/ceph/teuthology-archive/pdonnell-2020-02-15_16:51:06-fs-wip-pdonnell-testing-20200215.033325-distro-basic-smithi/4767980/teuthology.log

/a/nojha-2021-04-15_20:05:27-rados-wip-50217-distro-basic-smithi/6049676

/a/sseshasa-2021-05-17_11:08:21-rados-wip-sseshasa-testing-2021-05-17-1504-distro-basic-smithi/6118250

/a/yuriw-2021-06-28_17:32:48-rados-wip-yuri2-testing-2021-06-28-0858-pacific-distro-basic-smithi/6239590

/a/yuriw-2021-08-06_16:31:19-rados-wip-yuri-master-8.6.21-distro-basic-smithi/6324701

{
    "crash_id": "2021-08-26T03:38:46.109584Z_c0f5c111-a3bc-4210-8edd-e72cb5344590",
    "timestamp": "2021-08-26T03:38:46.109584Z",
    "process_name": "ceph-mon",
    "entity_name": "mon.c",
    "ceph_version": "v15.2.8.1.0.0",
    "utsname_hostname": "node-102",
    "utsname_sysname": "Linux",
    "utsname_release": "3.10.0-862.el7.x86_64",
    "utsname_version": "#1 SMP Fri Apr 20 16:44:24 UTC 2018",
    "utsname_machine": "x86_64",
    "os_name": "CentOS Linux",
    "os_id": "centos",
    "os_version_id": "7",
    "os_version": "7 (Core)",
    "assert_condition": "session_map.sessions.empty()",
    "assert_func": "virtual Monitor::~Monitor()",
    "assert_file": "/SDS-CICD/release/ceph15-tancz/rpmbuild/BUILD/ceph-15.2.8.1.0.0/src/mon/Monitor.cc",
    "assert_line": 262,
    "assert_thread_name": "ceph-mon",
    "assert_msg": "src/mon/Monitor.cc: In function 'virtual Monitor::~Monitor()' thread 7f8e02893340 time 2021-08-26T11:38:46.105871+0800\nsrc/mon/Monitor.cc: 262: FAILED ceph_assert(session_map.sessions.empty())\n",
    "backtrace": [
        "(()+0xf5d0) [0x7f8df78ce5d0]",
        "(gsignal()+0x37) [0x7f8df66c4207]",
        "(abort()+0x148) [0x7f8df66c58f8]",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19b) [0x7f8df9ac4c9e]",
        "(()+0x269e17) [0x7f8df9ac4e17]",
        "(Monitor::~Monitor()+0x846) [0x557a6eceded6]",
        "(Monitor::~Monitor()+0x9) [0x557a6ecedf29]",
        "(main()+0x260a) [0x557a6ec7ba9a]",
        "(__libc_start_main()+0xf5) [0x7f8df66b03d5]",
        "(()+0x2304f0) [0x557a6ecac4f0]" 
    ]
}

/a/sage-2021-10-28_02:19:01-rados-wip-sage3-testing-2021-10-27-1300-distro-basic-smithi/6464204

with logs!

/a/yuriw-2021-11-20_18:01:41-rados-wip-yuri8-testing-2021-11-20-0807-distro-basic-smithi/6516396