Ceph » cephsqlite

Nitzan Mordechai wrote:

/home/teuthworker/archive/yuriw-2022-04-29_15:44:49-rados-wip-yuri5-testing-2022-04-28-1007-distro-default-smithi/6813955

[...]

mgr log at the time of the crash:

2022-04-29T19:34:39.785+0000 7f5664339700 10 MonCommandCompletion::finish()
2022-04-29T19:34:39.785+0000 7f5664339700 20 mgr Gil Switched to new thread state 0x559d5f25c000
2022-04-29T19:34:39.785+0000 7f5664339700 20 mgr ~Gil Destroying new thread state 0x559d5f25c000
2022-04-29T19:34:39.785+0000 7f5664339700 10 mgr notify_all notify_all: notify_all command
2022-04-29T19:34:39.785+0000 7f5664339700 15 mgr notify_all queuing notify (command) to restful
2022-04-29T19:34:39.785+0000 7f5604101700  0 [devicehealth DEBUG root] skipping duplicate INTEL_SSDPEDMD400G4_CVFT623300GW400BGN
2022-04-29T19:34:39.785+0000 7f5604101700 10 ceph_option_get device_failure_prediction_mode found: none
2022-04-29T19:34:39.785+0000 7f5604101700  0 [devicehealth DEBUG root] set_kv('last_scrape', '20220429-193321')
2022-04-29T19:34:39.785+0000 7f56297f6700 20 mgr Gil Switched to new thread state 0x559d5ea22400
2022-04-29T19:34:39.785+0000 7f56297f6700 20 mgr ~Gil Destroying new thread state 0x559d5ea22400
2022-04-29T19:34:39.785+0000 7f5604101700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.y: unknown operation
2022-04-29T19:34:39.785+0000 7f5604101700 -1 devicehealth.serve:
2022-04-29T19:34:39.785+0000 7f5604101700 -1 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/devicehealth/module.py", line 376, in serve
    self.set_kv('last_scrape', last_scrape.strftime(TIME_FORMAT))
  File "/usr/share/ceph/mgr/mgr_module.py", line 1117, in set_kv
    self.db.execute(SQL, (key, value))
sqlite3.InternalError: unknown operation

Problem appears to be that libcephsqlite lost its RADOS lock:

2022-04-29T19:34:39.315+0000 7f5669343700  1 -- 172.21.15.155:0/832204322 <== osd.7 v1:172.21.15.169:6804/91796 154 ==== osd_op_reply(146 main.db.0000000000000000 [call] v59'104 uv103 ondisk = -2 ((2) No such file or directory)) v8 ==== 168+0+0 (unknown 3461770130 0 0) 0x559d5fa1d8c0 con 0x559d5f026000
2022-04-29T19:34:39.315+0000 7f55ef0d7700 -1 client.4184: SimpleRADOSStriper: lock_keeper_main: main.db: lock renewal failed: (2) No such file or directory

At this time, the mgr was noting some 10 pgs were inactive.

I will think about how we can make this mgr module more resilient to this type of situation.

Hi Patrick,

Can you please explain why unknown state of pgs results in this behavior?

The commonality we see between these issues so far is that they happen after an upgrade (on gibba and the LRC), or due to daemon restarts.

/a/yuriw-2022-12-07_15:47:33-rados-wip-yuri-testing-2022-12-06-1204-distro-default-smithi/7106555

I hit a quite similar problem. My cluster became unhealthy as follows.

```
$ kubectl exec n ${NS} deployment/rook-ceph-tools - ceph status
cluster:
id: b52d5f3d-ba14-442e-a089-0bca47b83758
health: HEALTH_ERR
441 large omap objects
Module 'devicehealth' has failed: unknown operation
1 osds down
1 host (1 osds) down
Degraded data redundancy: 1282/321355596 objects degraded (0.000%), 88 pgs degraded, 106 pgs undersized
62 pgs not deep-scrubbed in time
62 pgs not scrubbed in time
```

Before occurring this problem, the following message is shown in the mgr.a's log.

```
2023-03-16 09:08:24 debug 2023-03-16T00:08:24.478+0000 7feacbcf5700 -1 client.107899129: SimpleRADOSStriper: lock_keeper_main: main.db: lock renewal failed: (2) No such file or directory
```

At that time, mgr failover didn't happen. In addition, this problem was resolved after restarting mgr.a,

I have two questions about this.

a. Can my problem be considered to hit this issue's bug?
b. If a is true, is there any configuration to bypass this problem before 50291 is merged?

software environment:
- ceph: v17.2.5
- rook: v1.10.7