Ceph » CephFS

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:

 460  void FSMirror::mirror_status(Formatter *f) {
 461    std::scoped_lock locker(m_lock);
!462    f->open_object_section("status");
 463    if (m_init_failed) {
 464      f->dump_string("state", "failed");
 465    } else if (is_blocklisted(locker)) {
 466      f->dump_string("state", "blocklisted");
 467    } else {
 468      // dump rados addr for blocklist test
 469      f->dump_string("rados_inst", m_addrs);
 470      f->open_object_section("peers");
 471      for ([[maybe_unused]] auto &[peer, peer_replayer] : m_peer_replayers) {
 472        peer.dump(f);
 473      }
─── Variables ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
arg this = 0x5601b6cfe000: {m_cct = 0x5601b1e9a000,m_filesystem = {fscid = 1,fs_name = "a"},m_pool_id …, f = 0x5601b6d28e00: {_vptr.Formatter = 0x7fb7cf694b30 <vtable for ceph::JSONFormatter+16>}
loc locker = {_M_device = @0x5601b6cfe060}
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>>> p m_init_failed
$4 = false

The command:

./bin/ceph --admin-daemon asok/client.admin.342404.asok fs mirror status a@1

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:
[...]

The command:
[...]
works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Do you have the run link? I'll have a look now.

Venky Shankar wrote:

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:
[...]

The command:
[...]
works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Do you have the run link? I'll have a look now.

The above finding is from the vstart cluster reproduced.
I was also looking at https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/7313862.

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

Venky Shankar wrote:

Jos Collin wrote:

@Venky,

When we disable mirroring using 'ceph mgr module disable mirroring' and then enable using 'ceph fs mirror enable <primary_fs_name>', the 'fs mirror status' command doesn't output the "state: failed" status. The mirroring seems failed, but the "state: failed" doesn't show.

It doesn't hit 'FSMirror::init', so that m_init_failed is set to True.

So, some change likely broke the part where m_init_failed gets set or there is a delayed check for m_init_failed. I would start with the latter and hash that out first.

It doesn't seem to be a delayed check for m_init_failed. The debug always shows m_init_failed = false, even after waiting for a few minutes:
[...]

The command:
[...]
works, but it doesn't show the 'state'. So this should be a problem with failed to set m_init_failed value. Checking that.

Do you have the run link? I'll have a look now.

The above finding is from the vstart cluster reproduced.
I was also looking at https://pulpito.ceph.com/rishabh-2023-06-23_17:37:30-fs-wip-rishabh-improvements-authmon-distro-default-smithi/7313862.

No problem. Just upload the cephfs-mirror debug logs from the vstart cluster when running this test.

Building the failed branch again, as none of these qa runs mentioned in the Description is having remote logs.

Jos Collin wrote in #note-12:

test_mirroring_init_failure_with_recovery make the mirror daemon fail. After this, the mirror daemon didn't restarted to show the 'state' == 'failed' status. This is confirmed by the error message in the description and the subsequent 'mirror status' command shows 'Commond failed' in teuthology.log. This issue is being fixed by PR#56193.

Do you mean the FSMirror instance didn't restart or the mirror daemon crashed?

Venky Shankar wrote in #note-13:

Jos Collin wrote in #note-12:

test_mirroring_init_failure_with_recovery make the mirror daemon fail. After this, the mirror daemon didn't restarted to show the 'state' 'failed' status. This is confirmed by the error message in the description and the subsequent 'mirror status' command shows 'Commond failed' in teuthology.log. This issue is being fixed by PR#56193.

Do you mean the FSMirror instance didn't restart or the mirror daemon crashed?

In test_mirroring_init_failure_with_recovery, as I understand, we are failing the mirror daemon. Then it should restart after 30 seconds and 'state' 'failed' status is checked in the test. Because of the time check bug (fixed by https://github.com/ceph/ceph/pull/56193/commits/518fe18c0220bbe755df7a80db040c18667765a8), the mirror daemon is not restarting. There's no crash noted in the logs, but it fails to restart. Thus this tracker is also referred in PR#56193.

Jos Collin wrote in #note-14:

Venky Shankar wrote in #note-13:

Jos Collin wrote in #note-12:

test_mirroring_init_failure_with_recovery make the mirror daemon fail. After this, the mirror daemon didn't restarted to show the 'state' 'failed' status. This is confirmed by the error message in the description and the subsequent 'mirror status' command shows 'Commond failed' in teuthology.log. This issue is being fixed by PR#56193.

Do you mean the FSMirror instance didn't restart or the mirror daemon crashed?

In test_mirroring_init_failure_with_recovery, as I understand, we are failing the mirror daemon. Then it should restart after 30 seconds and 'state' 'failed' status is checked in the test. Because of the time check bug (fixed by https://github.com/ceph/ceph/pull/56193/commits/518fe18c0220bbe755df7a80db040c18667765a8), the mirror daemon is not restarting. There's no crash noted in the logs, but it fails to restart. Thus this tracker is also referred in PR#56193.

kk, thanks.