Project

General

Profile

Actions

Bug #74149

closed

Prometheus module fails when trying to load security configuration JSON

Added by Laura Flores 3 months ago. Updated 16 days ago.

Status:
Resolved
Priority:
Normal
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Fixed In:
v20.3.0-5831-g7d9f8f3b5f
Released In:
Upkeep Timestamp:
2026-03-05T06:50:36+00:00

Description

/a/teuthology-2025-12-07_20:00:23-rados-main-distro-default-smithi/8644698

2025-12-07T22:20:21.297 INFO:teuthology.orchestra.run.smithi099.stderr:2025-12-07T22:20:21.290+0000 7fd8d062a640  1 -- 172.21.15.99:0/1116136488 wait complete.
2025-12-07T22:20:21.315 INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_prometheus.TestPrometheus) ... FAIL
2025-12-07T22:20:21.315 INFO:tasks.cephfs_test_runner:
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:======================================================================
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:FAIL: test_standby (tasks.mgr.test_prometheus.TestPrometheus)
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_6ce249e0e13e12a74d5c855ed12d6b50671977c9/qa/tasks/mgr/test_prometheus.py", line 50, in test_standby
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:    self.assertEqual(r.headers["content-type"], "text/plain;charset=utf-8")
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:AssertionError: 'text/html;charset=utf-8' != 'text/plain;charset=utf-8'
2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:- text/html;charset=utf-8
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:?      ^^^
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:+ text/plain;charset=utf-8
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:?      ^ +++
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:Ran 2 tests in 122.270s
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:
2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:FAILED (failures=1)

/a/teuthology-2025-12-07_20:00:23-rados-main-distro-default-smithi/8644698/remote/smithi099/log/ceph-mgr.x.log.gz

2025-12-07T22:19:14.340+0000 7fdce73cb640 20 mgr.server operator() health checks:
{}

2025-12-07T22:19:14.340+0000 7fdce73cb640  0 log_channel(cluster) log [DBG] : pgmap v3: 105 pgs: 105 active+clean; 584 KiB data, 108 MiB used, 360 GiB / 360 GiB avail
2025-12-07T22:19:14.340+0000 7fdce73cb640  1 -- 172.21.15.99:0/3828570172 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- monmgrreport(gid 4820, 0 checks, 0 progress events) -- 0x5608cc912240 con 0x5608c5881000
2025-12-07T22:19:14.348+0000 7fdcdbab4640  0 [prometheus ERROR root] Failed to setup cephadm based secure monitoring stack: Expecting value: line 1 column 1 (char 0)
Falling back to default configuration
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/prometheus/module.py", line 1942, in configure
    security_config = json.loads(out)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I suspect this PR: https://github.com/ceph/ceph/pull/65245


Related issues 1 (1 open0 closed)

Related to mgr - Bug #74148: Prometheus module experiences connection issues related to cherrypyPending BackportNitzan Mordechai

Actions
Actions #1

Updated by Laura Flores 3 months ago

  • Related to Bug #74148: Prometheus module experiences connection issues related to cherrypy added
Actions #2

Updated by Aishwarya Mathuria 3 months ago

/a/yuriw-2025-12-03_15:44:36-rados-wip-yuri5-testing-2025-12-02-1256-distro-default-smithi/8639576

2025-12-03T16:53:11.873 INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_prometheus.TestPrometheus) ... FAIL
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:======================================================================
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:FAIL: test_standby (tasks.mgr.test_prometheus.TestPrometheus)
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_151fc19e8957de33a9ab329f5cd67d0d2eab7212/qa/tasks/mgr/test_prometheus.py", line 50, in test_standby
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:    self.assertEqual(r.headers["content-type"], "text/plain;charset=utf-8")
2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:AssertionError: 'text/html;charset=utf-8' != 'text/plain;charset=utf-8'
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:- text/html;charset=utf-8
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:?      ^^^
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:+ text/plain;charset=utf-8
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:?      ^ +++
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:Ran 2 tests in 121.602s

Actions #3

Updated by Kamoltat (Junior) Sirivadhna 3 months ago

rados suite watcher:
ping @Nitzan Mordechai

Actions #4

Updated by Nitzan Mordechai 3 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 66571
Actions #5

Updated by Laura Flores about 2 months ago

/a/nmordech-2026-01-25_11:10:14-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/16984

Actions #6

Updated by Laura Flores about 2 months ago

/a/lflores-2026-01-26_23:21:06-rados-wip-yuri12-testing-2026-01-22-2045-distro-default-trial/19108

Actions #7

Updated by Sridhar Seshasayee about 2 months ago

/a/skanta-2026-01-27_05:35:03-rados-wip-bharath1-testing-2026-01-26-1242-distro-default-trial/19777

Actions #8

Updated by Aishwarya Mathuria about 1 month ago

/a/skanta-2026-01-30_23:46:16-rados-wip-bharath7-testing-2026-01-29-2016-distro-default-trial/28581

Actions #9

Updated by Laura Flores about 1 month ago

/a/yuriw-2026-02-03_16:00:06-rados-wip-yuri4-testing-2026-02-02-2122-distro-default-trial/31894

Actions #10

Updated by Aishwarya Mathuria about 1 month ago

/a/skanta-2026-02-07_00:02:26-rados-wip-bharath7-testing-2026-02-06-0906-distro-default-trial/39128

Actions #11

Updated by Aishwarya Mathuria about 1 month ago

/a/skanta-2026-02-05_03:38:32-rados-wip-bharath2-testing-2026-02-03-0542-distro-default-trial/35664

Actions #12

Updated by Aishwarya Mathuria about 1 month ago ยท Edited

Seen in squid: /a/yuriw-2026-02-17_20:43:43-rados-wip-yuri6-testing-2026-02-17-1732-squid-distro-default-trial/53893 because https://github.com/ceph/ceph/pull/66483 was included in the QA batch by mistake.

Actions #13

Updated by Upkeep Bot 16 days ago

  • Status changed from Fix Under Review to Resolved
  • Merge Commit set to 7d9f8f3b5f2112299079105c5582c6208348002d
  • Fixed In set to v20.3.0-5831-g7d9f8f3b5f
  • Upkeep Timestamp set to 2026-03-05T06:50:36+00:00
Actions

Also available in: Atom PDF