Bug #74149
closedPrometheus module fails when trying to load security configuration JSON
0%
Description
/a/teuthology-2025-12-07_20:00:23-rados-main-distro-default-smithi/8644698
2025-12-07T22:20:21.297 INFO:teuthology.orchestra.run.smithi099.stderr:2025-12-07T22:20:21.290+0000 7fd8d062a640 1 -- 172.21.15.99:0/1116136488 wait complete. 2025-12-07T22:20:21.315 INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_prometheus.TestPrometheus) ... FAIL 2025-12-07T22:20:21.315 INFO:tasks.cephfs_test_runner: 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:====================================================================== 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:FAIL: test_standby (tasks.mgr.test_prometheus.TestPrometheus) 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_6ce249e0e13e12a74d5c855ed12d6b50671977c9/qa/tasks/mgr/test_prometheus.py", line 50, in test_standby 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner: self.assertEqual(r.headers["content-type"], "text/plain;charset=utf-8") 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:AssertionError: 'text/html;charset=utf-8' != 'text/plain;charset=utf-8' 2025-12-07T22:20:21.316 INFO:tasks.cephfs_test_runner:- text/html;charset=utf-8 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:? ^^^ 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:+ text/plain;charset=utf-8 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:? ^ +++ 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner: 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner: 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:Ran 2 tests in 122.270s 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner: 2025-12-07T22:20:21.317 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
/a/teuthology-2025-12-07_20:00:23-rados-main-distro-default-smithi/8644698/remote/smithi099/log/ceph-mgr.x.log.gz
2025-12-07T22:19:14.340+0000 7fdce73cb640 20 mgr.server operator() health checks:
{}
2025-12-07T22:19:14.340+0000 7fdce73cb640 0 log_channel(cluster) log [DBG] : pgmap v3: 105 pgs: 105 active+clean; 584 KiB data, 108 MiB used, 360 GiB / 360 GiB avail
2025-12-07T22:19:14.340+0000 7fdce73cb640 1 -- 172.21.15.99:0/3828570172 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- monmgrreport(gid 4820, 0 checks, 0 progress events) -- 0x5608cc912240 con 0x5608c5881000
2025-12-07T22:19:14.348+0000 7fdcdbab4640 0 [prometheus ERROR root] Failed to setup cephadm based secure monitoring stack: Expecting value: line 1 column 1 (char 0)
Falling back to default configuration
Traceback (most recent call last):
File "/usr/share/ceph/mgr/prometheus/module.py", line 1942, in configure
security_config = json.loads(out)
File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I suspect this PR: https://github.com/ceph/ceph/pull/65245
Updated by Laura Flores 3 months ago
- Related to Bug #74148: Prometheus module experiences connection issues related to cherrypy added
Updated by Aishwarya Mathuria 3 months ago
/a/yuriw-2025-12-03_15:44:36-rados-wip-yuri5-testing-2025-12-02-1256-distro-default-smithi/8639576
2025-12-03T16:53:11.873 INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_prometheus.TestPrometheus) ... FAIL 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner: 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:====================================================================== 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:FAIL: test_standby (tasks.mgr.test_prometheus.TestPrometheus) 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_151fc19e8957de33a9ab329f5cd67d0d2eab7212/qa/tasks/mgr/test_prometheus.py", line 50, in test_standby 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner: self.assertEqual(r.headers["content-type"], "text/plain;charset=utf-8") 2025-12-03T16:53:11.874 INFO:tasks.cephfs_test_runner:AssertionError: 'text/html;charset=utf-8' != 'text/plain;charset=utf-8' 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:- text/html;charset=utf-8 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:? ^^^ 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:+ text/plain;charset=utf-8 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:? ^ +++ 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner: 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner: 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2025-12-03T16:53:11.875 INFO:tasks.cephfs_test_runner:Ran 2 tests in 121.602s
Updated by Kamoltat (Junior) Sirivadhna 3 months ago
rados suite watcher:
ping @Nitzan Mordechai
Updated by Nitzan Mordechai 3 months ago
- Status changed from New to Fix Under Review
- Pull request ID set to 66571
Updated by Laura Flores about 2 months ago
/a/nmordech-2026-01-25_11:10:14-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/16984
Updated by Laura Flores about 2 months ago
/a/lflores-2026-01-26_23:21:06-rados-wip-yuri12-testing-2026-01-22-2045-distro-default-trial/19108
Updated by Sridhar Seshasayee about 2 months ago
/a/skanta-2026-01-27_05:35:03-rados-wip-bharath1-testing-2026-01-26-1242-distro-default-trial/19777
Updated by Aishwarya Mathuria about 1 month ago
/a/skanta-2026-01-30_23:46:16-rados-wip-bharath7-testing-2026-01-29-2016-distro-default-trial/28581
Updated by Laura Flores about 1 month ago
/a/yuriw-2026-02-03_16:00:06-rados-wip-yuri4-testing-2026-02-02-2122-distro-default-trial/31894
Updated by Aishwarya Mathuria about 1 month ago
/a/skanta-2026-02-07_00:02:26-rados-wip-bharath7-testing-2026-02-06-0906-distro-default-trial/39128
Updated by Aishwarya Mathuria about 1 month ago
/a/skanta-2026-02-05_03:38:32-rados-wip-bharath2-testing-2026-02-03-0542-distro-default-trial/35664
Updated by Aishwarya Mathuria about 1 month ago ยท Edited
Seen in squid: /a/yuriw-2026-02-17_20:43:43-rados-wip-yuri6-testing-2026-02-17-1732-squid-distro-default-trial/53893 because https://github.com/ceph/ceph/pull/66483 was included in the QA batch by mistake.
Updated by Upkeep Bot 16 days ago
- Status changed from Fix Under Review to Resolved
- Merge Commit set to 7d9f8f3b5f2112299079105c5582c6208348002d
- Fixed In set to v20.3.0-5831-g7d9f8f3b5f
- Upkeep Timestamp set to 2026-03-05T06:50:36+00:00