Ceph » RADOS

@Kamoltat (Junior) Sirivadhna can you attach the relevant BZ?

scrub note: bump up. Need to known the details.

https://bugzilla.redhat.com/show_bug.cgi?id=2174745

@Kamoltat (Junior) Sirivadhna i couldn't find any "resend forwarded message to leader" events in the logs that the BZ attached, maybe I'm missing something?

the 2 op's that showing as stuck are "forward_request_leader" that never come back. I'll keep checking those logs, but if you have any hints, please let me know

@Nitzan Mordechai
So I saw in the BZ that two slow ops events were `forward_request_leader`and `forwarded`. To my understanding, I thought that because the mon gets stuck when they `forwarded` the request to the leader, it is endlessly waiting for something to come back so that it can take action with its current operation. However, nothing came back it seems not to drop the request, therefore, I think the problem in the BZ is similar to that of the tracker, the only difference is that the tracker's `resend forwarded message to leader` as opposed to `forwarded`, but in my opinion they do share a common problem.

@Kamoltat (Junior) Sirivadhna thanks for the info! will check that

@Nitzan Mordechai I think this BZ downstream might be more relevant,
since the last event has `resend forwarded message to leader`

{
    "ops": [
        {
            "description": "log(1 entries from seq 2 at 2024-05-30T06:54:15.543263+0000)",
            "initiated_at": "2024-05-30T06:54:15.907602+0000",
            "age": 3723.1818099669999,
            "duration": 3723.1818310100002,
            "continuous": false,
            "type_data": {
                "events": [
                    {
                        "event": "initiated",
                        "time": "2024-05-30T06:54:15.907602+0000",
                        "duration": 3.4299999999999999e-07
                    },
                    {
                        "event": "throttled",
                        "time": "2024-05-30T06:54:15.907602+0000",
                        "duration": 4294967295.9999995
                    },
                    {
                        "event": "header_read",
                        "time": "2024-05-30T06:54:15.907602+0000",
                        "duration": 1.0716999999999999e-05
                    },
                    {
                        "event": "all_read",
                        "time": "2024-05-30T06:54:15.907612+0000",
                        "duration": 0.00068134100000000002
                    },
                    {
                        "event": "dispatched",
                        "time": "2024-05-30T06:54:15.908294+0000",
                        "duration": 8.9400000000000004e-07
                    },
                    {
                        "event": "mon:_ms_dispatch",
                        "time": "2024-05-30T06:54:15.908295+0000",
                        "duration": 1.6400000000000001e-07
                    },
                    {
                        "event": "mon:dispatch_op",
                        "time": "2024-05-30T06:54:15.908295+0000",
                        "duration": 3.5900000000000003e-07
                    },
                    {
                        "event": "psvc:dispatch",
                        "time": "2024-05-30T06:54:15.908295+0000",
                        "duration": 8.5199999999999997e-06
                    },
                    {
                        "event": "logm:preprocess_query",
                        "time": "2024-05-30T06:54:15.908304+0000",
                        "duration": 7.2600000000000002e-07
                    },
                    {
                        "event": "logm:preprocess_log",
                        "time": "2024-05-30T06:54:15.908304+0000",
                        "duration": 1.2514000000000001e-05
                    },
                    {
                        "event": "forward_request_leader",
                        "time": "2024-05-30T06:54:15.908317+0000",
                        "duration": 1.0344e-05
                    },
                    {
                        "event": "forwarded",
                        "time": "2024-05-30T06:54:15.908327+0000",
                        "duration": 5.3054240999999998
                    },
                    {
                        "event": "resend forwarded message to leader",
                        "time": "2024-05-30T06:54:21.213751+0000",
                        "duration": 5.3061496789999998
                    }
                ],
                "info": {
                    "seq": 136274,
                    "src_is_mon": false,
                    "source": "osd.21 v2:10.0.210.182:6808/1297045346",
                    "forwarded_to_leader": true
                }
            }
        }
    ],
    "num_ops": 1
}

@Kamoltat (Junior) Sirivadhna from where did you get that log snap ? maybe I'm missing some of those logs

@Kamoltat (Junior) Sirivadhna looking at this from bug scrub: any info on the above comment?

https://bugzilla.redhat.com/show_bug.cgi?id=2283916 - Observing slow_ops on a mon daemon post site down tests in a 3 AZ cluster
https://bugzilla.redhat.com/show_bug.cgi?id=2174745 - [MDR][Stretch Cluster] Observing Slow Ops on one monitor post OSD replacement in a Stretch cluster

@Nitzan Mordechai: ping.

Bug scrub note: still need more info

Bump up. @Nitzan Mordechai

still was not able to reproduce, working on it

scrub note: approved, sent to QA.

scrub note: in QA.