Project

General

Profile

Actions
Copy link

Bug #64968

closed

mon: MON_DOWN warnings when mons are first booting

Added by Patrick Donnelly about 2 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Patrick Donnelly
Category:
Administration/Usability
Target version:
Ceph - v20.0.0
% Done:

0%

Source:
Q/A
Backport:
squid,reef
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor
Pull request ID:
56271
Tags (freeform):
Merge Commit:
2716b6e76ee43c6ac6cb353cf44fd0ecb95cb620
Fixed In:
v19.3.0-824-g2716b6e76e
Released In:
v20.2.0~3283
Upkeep Timestamp:
2025-11-20T01:23:34+00:00

Description

2024-03-18T00:42:36.045 INFO:teuthology.orchestra.run.smithi162.stdout:Creating initial monmap...
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1b79961c-e4c0-11ee-95c9-87774f69a715
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:monmaptool for a [v2:172.21.15.162:3300,v1:172.21.15.162:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:setting min_mon_release = quincy
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: set fsid to 1b79961c-e4c0-11ee-95c9-87774f69a715
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:
...
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:Creating mon...
2024-03-18T00:42:37.841 INFO:teuthology.orchestra.run.smithi162.stdout:create mon.a on
...
2024-03-18T00:44:20.339 INFO:tasks.cephadm:Adding mon.a on smithi162
2024-03-18T00:44:20.339 INFO:tasks.cephadm:Adding mon.b on smithi179
2024-03-18T00:44:20.339 INFO:tasks.cephadm:Adding mon.c on smithi181
...
2024-03-18T00:44:23.616 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:23 smithi162 ceph-mon[32185]: Deploying daemon mon.c on smithi181
...
2024-03-18T00:44:33.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 18 00:44:32 smithi181 ceph-mon[35664]: Deploying daemon mon.b on smithi179
2024-03-18T00:44:33.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 18 00:44:32 smithi181 ceph-mon[35664]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2024-03-18T00:44:33.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 18 00:44:32 smithi181 ceph-mon[35664]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x'
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: Deploying daemon mon.b on smithi179
...
2024-03-18T00:44:41.117 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:40 smithi162 ceph-mon[32185]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)
2024-03-18T00:44:41.117 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:40 smithi162 ceph-mon[32185]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c
2024-03-18T00:44:41.117 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:40 smithi162 ceph-mon[32185]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c

From: /teuthology/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608542/teuthology.log

Related issues 2 (0 open — 2 closed)

Copied to RADOS - Backport #65081: squid: mon: MON_DOWN warnings when mons are first bootingResolvedPatrick DonnellyActions
Copied to RADOS - Backport #65082: reef: mon: MON_DOWN warnings when mons are first bootingResolvedPatrick DonnellyActions
Actions
Copy link
 #1

Updated by Patrick Donnelly about 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 56271
Actions
Copy link
 #2

Updated by Patrick Donnelly about 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #3

Updated by Upkeep Bot about 2 years ago

  • Copied to Backport #65081: squid: mon: MON_DOWN warnings when mons are first booting added
Actions
Copy link
 #4

Updated by Upkeep Bot about 2 years ago

  • Copied to Backport #65082: reef: mon: MON_DOWN warnings when mons are first booting added
Actions
Copy link
 #6

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to 2716b6e76ee43c6ac6cb353cf44fd0ecb95cb620
  • Fixed In set to v19.3.0-824-g2716b6e76ee
  • Upkeep Timestamp set to 2025-07-09T13:47:08+00:00
Actions
Copy link
 #7

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-824-g2716b6e76ee to v19.3.0-824-g2716b6e76e
  • Upkeep Timestamp changed from 2025-07-09T13:47:08+00:00 to 2025-07-14T17:40:57+00:00
Actions
Copy link
 #8

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~3283
  • Upkeep Timestamp changed from 2025-07-14T17:40:57+00:00 to 2025-11-01T00:55:11+00:00
Actions
Copy link
 #9

Updated by Upkeep Bot 4 months ago

  • Status changed from Pending Backport to Resolved
  • Upkeep Timestamp changed from 2025-11-01T00:55:11+00:00 to 2025-11-20T01:23:34+00:00
Actions
Copy link

Also available in: Atom PDF