Project

General

Profile

Actions

Bug #68747

closed

fs:upgrade failure due to ceph-mgr not being ready

Added by Venky Shankar over 1 year ago. Updated 6 months ago.

Status:
Duplicate
Priority:
Normal
Category:
Administration/Usability
Target version:
% Done:

0%

Source:
Q/A
Backport:
quincy,reef,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

/a/vshankar-2024-10-17_06:38:16-fs-wip-vshankar-testing-20241016.135728-debug-testing-default-smithi/7953019

2024-10-18T06:52:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-10-18T06:52:14.590+0000 7fa8c2000640  1 -- 172.21.15.29:0/2607670694 <== mgr.24435 v2:172.21.15.60:6828/3675116603 1 ==== mgr_command_reply(tid 0: -95 Warning: due to
ceph-mgr restart, some PG states may not be up to date
2024-10-18T06:52:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it) ==== 222+0+0 (secure 0 0 0) 0x7fa8ac009030 con 0x7fa8a401ed50
2024-10-18T06:52:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
2024-10-18T06:52:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it
2024-10-18T06:52:15.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e91633d8-8d1b-11ef-bb99-d5e06f7e0c9a -e sha1=7013ef110132885a91981a9f86f5936deb441686 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"''

ceph-mgr isn't ready to process requests right after it got upgraded. So, maybe, the test needs to wait a bit before making requests to ceph-mgr or the ceph-mgr should declare itself available only after all plugins are ready otherwise. I think this is similar to what @Milind Changire ran into a while ago.


Related issues 1 (1 open0 closed)

Related to CephFS - Bug #67230: mgr: should be declared available only after all python modules have been loadedFix Under ReviewMahesh Mohan

Actions
Actions #1

Updated by Venky Shankar over 1 year ago

  • Status changed from New to Triaged
  • Assignee set to Milind Changire

Milind, please link the tracker that tracks the ceph-mgr issue and mark this as a duplicate.

Actions #2

Updated by Milind Changire over 1 year ago

  • Status changed from Triaged to Duplicate
Actions #3

Updated by Venky Shankar over 1 year ago

Another instance - https://pulpito.ceph.com/vshankar-2024-10-30_07:41:03-fs-wip-vshankar-testing-20241029.045257-debug-testing-default-smithi/7973384/

Why is this issue seen frequently now? Did something change in the manager that increases the likelihood of running into this?

Actions #4

Updated by Milind Changire over 1 year ago

Venky Shankar wrote in #note-3:

Another instance - https://pulpito.ceph.com/vshankar-2024-10-30_07:41:03-fs-wip-vshankar-testing-20241029.045257-debug-testing-default-smithi/7973384/

Why is this issue seen frequently now? Did something change in the manager that increases the likelihood of running into this?

Nothing changed from my side.

Actions #5

Updated by Venky Shankar over 1 year ago

  • Related to Bug #67230: mgr: should be declared available only after all python modules have been loaded added
Actions #6

Updated by Venky Shankar over 1 year ago

Milind Changire wrote in #note-4:

Venky Shankar wrote in #note-3:

Another instance - https://pulpito.ceph.com/vshankar-2024-10-30_07:41:03-fs-wip-vshankar-testing-20241029.045257-debug-testing-default-smithi/7973384/

Why is this issue seen frequently now? Did something change in the manager that increases the likelihood of running into this?

Nothing changed from my side.

Oh, sure thing. I was just wondering if some recent commit made this show up frequently :)

Actions #8

Updated by Venky Shankar about 1 year ago

  • Related to Bug #69899: Test failure: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) added
Actions #9

Updated by Venky Shankar about 1 year ago

  • Related to deleted (Bug #69899: Test failure: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS))
Actions

Also available in: Atom PDF