Project

General

Profile

Actions
Copy link

Bug #64970

closed

mgr/dashboard: fix duplicate grafana panels when on mgr failover

Added by Avan Thakkar about 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Avan Thakkar
Category:
Monitoring
Target version:
-
% Done:

0%

Source:
Backport:
quincy, reef, squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
56626
Tags (freeform):
Merge Commit:
a6549451253d26216d353246c0fe6f6d80f3b9cf
Fixed In:
v19.3.0-1570-ga654945125
Released In:
v20.2.0~3105
Upkeep Timestamp:
2025-11-01T01:19:40+00:00

Description

Description of problem

here

The metrics data was not shown with "multiple matches for labels grouping labels must ensure unique matches" messages on Grafana dashboard after active mgr failover.

This is a screenshot(*1) of "ceph_pool_rd_bytes" data before failove the active mgr. The only one instance(10.74.214.226 - Active mgr then) is shown.
(*1) please see 2024-02-07 14-47-25_before_failover_the_mgr.png

But after active mgr failover, 2 instances (10.74.214.226 and mons-2.cephtest61.lab.psi.pnq2.redhat.com - Current Active mgr) are shown(*2)
(*2) please see 2024-02-07 14-47-32_after_failover_the_mgr.png

It seems there are 2 instances with same metrics collected by ceph-mgr after mgr failover
(one is the metrics collected by previous active mgr, Another is the metrics currently collecting by current active mgr)
Grafana tried to display it, but It failed with "multiple matches for labels grouping labels must ensure unique matches" message.

Files

Download all files
Screenshot from 2024-03-18 23-12-14.png (215 KB) Screenshot from 2024-03-18 23-12-14.png after_failover_the_mgr Avan Thakkar, 03/18/2024 05:43 PM
Screenshot from 2024-03-18 23-12-47.png (313 KB) Screenshot from 2024-03-18 23-12-47.png before_failover_the_mgr Avan Thakkar, 03/18/2024 05:43 PM
Screenshot from 2024-03-18 23-12-07.png (200 KB) Screenshot from 2024-03-18 23-12-07.png multiple matches issue grafan panel Avan Thakkar, 03/18/2024 05:43 PM

Related issues 3 (0 open — 3 closed)

Copied to Dashboard - Backport #65512: reef: mgr/dashboard: fix duplicate grafana panels when on mgr failoverResolvedAdam KingActions
Copied to Dashboard - Backport #65513: quincy: mgr/dashboard: fix duplicate grafana panels when on mgr failoverResolvedAdam KingActions
Copied to Dashboard - Backport #65515: squid: mgr/dashboard: fix duplicate grafana panels when on mgr failoverResolvedAdam KingActions
Actions
Copy link
 #1

Updated by Avan Thakkar almost 2 years ago

  • Backport changed from quincy, reef to quincy, reef, squid
Actions
Copy link
 #2

Updated by Avan Thakkar almost 2 years ago

  • Pull request ID set to 56626
Actions
Copy link
 #3

Updated by Adam King almost 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #4

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #65512: reef: mgr/dashboard: fix duplicate grafana panels when on mgr failover added
Actions
Copy link
 #5

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #65513: quincy: mgr/dashboard: fix duplicate grafana panels when on mgr failover added
Actions
Copy link
 #6

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #65515: squid: mgr/dashboard: fix duplicate grafana panels when on mgr failover added
Actions
Copy link
 #8

Updated by Adam King over 1 year ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #9

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to a6549451253d26216d353246c0fe6f6d80f3b9cf
  • Fixed In set to v19.3.0-1570-ga6549451253
  • Upkeep Timestamp set to 2025-07-11T14:14:41+00:00
Actions
Copy link
 #10

Updated by Upkeep Bot 9 months ago

  • Fixed In changed from v19.3.0-1570-ga6549451253 to v19.3.0-1570-ga654945125
  • Upkeep Timestamp changed from 2025-07-11T14:14:41+00:00 to 2025-07-14T23:10:12+00:00
Actions
Copy link
 #11

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~3105
  • Upkeep Timestamp changed from 2025-07-14T23:10:12+00:00 to 2025-11-01T01:19:40+00:00
Actions
Copy link

Also available in: Atom PDF