Bug #68747
closedfs:upgrade failure due to ceph-mgr not being ready
0%
Description
/a/vshankar-2024-10-17_06:38:16-fs-wip-vshankar-testing-20241016.135728-debug-testing-default-smithi/7953019
2024-10-18T06:52:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-10-18T06:52:14.590+0000 7fa8c2000640 1 -- 172.21.15.29:0/2607670694 <== mgr.24435 v2:172.21.15.60:6828/3675116603 1 ==== mgr_command_reply(tid 0: -95 Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-10-18T06:52:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it) ==== 222+0+0 (secure 0 0 0) 0x7fa8ac009030 con 0x7fa8a401ed50 2024-10-18T06:52:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-10-18T06:52:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-10-18T06:52:15.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e91633d8-8d1b-11ef-bb99-d5e06f7e0c9a -e sha1=7013ef110132885a91981a9f86f5936deb441686 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"''
ceph-mgr isn't ready to process requests right after it got upgraded. So, maybe, the test needs to wait a bit before making requests to ceph-mgr or the ceph-mgr should declare itself available only after all plugins are ready otherwise. I think this is similar to what @Milind Changire ran into a while ago.
Updated by Venky Shankar over 1 year ago
- Status changed from New to Triaged
- Assignee set to Milind Changire
Milind, please link the tracker that tracks the ceph-mgr issue and mark this as a duplicate.
Updated by Milind Changire over 1 year ago
- Status changed from Triaged to Duplicate
PR - https://github.com/ceph/ceph/pull/59089
DUPLICATE of: https://tracker.ceph.com/issues/67230
Updated by Venky Shankar over 1 year ago
Another instance - https://pulpito.ceph.com/vshankar-2024-10-30_07:41:03-fs-wip-vshankar-testing-20241029.045257-debug-testing-default-smithi/7973384/
Why is this issue seen frequently now? Did something change in the manager that increases the likelihood of running into this?
Updated by Milind Changire over 1 year ago
Venky Shankar wrote in #note-3:
Another instance - https://pulpito.ceph.com/vshankar-2024-10-30_07:41:03-fs-wip-vshankar-testing-20241029.045257-debug-testing-default-smithi/7973384/
Why is this issue seen frequently now? Did something change in the manager that increases the likelihood of running into this?
Nothing changed from my side.
Updated by Venky Shankar over 1 year ago
- Related to Bug #67230: mgr: should be declared available only after all python modules have been loaded added
Updated by Venky Shankar over 1 year ago
Milind Changire wrote in #note-4:
Venky Shankar wrote in #note-3:
Another instance - https://pulpito.ceph.com/vshankar-2024-10-30_07:41:03-fs-wip-vshankar-testing-20241029.045257-debug-testing-default-smithi/7973384/
Why is this issue seen frequently now? Did something change in the manager that increases the likelihood of running into this?
Nothing changed from my side.
Oh, sure thing. I was just wondering if some recent commit made this show up frequently :)
Updated by Venky Shankar about 1 year ago
- Related to Bug #69899: Test failure: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS) added
Updated by Venky Shankar about 1 year ago
- Related to deleted (Bug #69899: Test failure: test_exports_on_mgr_restart (tasks.cephfs.test_nfs.TestNFS))
Updated by Dhairya Parmar 6 months ago
tentacle:
https://pulpito.ceph.com/jcollin-2025-08-25_02:17:45-fs:upgrade-wip-jcollin-testing-20250824.075829-tentacle-distro-default-smithi/8462353/
https://pulpito.ceph.com/jcollin-2025-08-25_02:17:45-fs:upgrade-wip-jcollin-testing-20250824.075829-tentacle-distro-default-smithi/8462400/
https://pulpito.ceph.com/jcollin-2025-08-25_02:17:45-fs:upgrade-wip-jcollin-testing-20250824.075829-tentacle-distro-default-smithi/8462370/