Bug #74980
openPort 7789 still in use, waiting...
0%
Description
/a/nmordech-2026-02-17_05:46:21-rados-wip-rocky10-branch-of-the-day-2026-02-13-1771001916-distro-default-trial/51932
/a/nmordech-2026-02-17_05:46:21-rados-wip-rocky10-branch-of-the-day-2026-02-13-1771001916-distro-default-trial/51710
2026-02-17T06:19:53.400+0000 7fe1dbbf46c0 10 mgr notify_all notify_all: notify_all command
2026-02-17T06:19:53.400+0000 7fe0ffd0c6c0 0 [prometheus DEBUG prometheus] mon_command: 'orch get-security-config' -> 0 in 0.001s
2026-02-17T06:19:53.400+0000 7fe0ffd0c6c0 0 [prometheus ERROR prometheus] Failed to setup cephadm based secure monitoring stack: Expecting value: line 1 column 1 (char 0)
Falling back to default configuration
Traceback (most recent call last):
File "/usr/share/ceph/mgr/prometheus/module.py", line 2017, in configure
security_config = json.loads(out)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/json/decoder.py", line 338, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/json/decoder.py", line 356, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2026-02-17T06:19:53.401+0000 7fe0ffd0c6c0 0 [prometheus DEBUG prometheus] Port 7789 still in use, waiting...
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 10 cephx: validate_tickets want 55 have 55 need 0
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 20 cephx client: need_tickets: want=55 have=55 need=0
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 10 auth: dump_rotating:
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 10 auth: id 1 AQD5B5Rp8CDQHhAA6tEfIEYfOJ2H7YixUsTdhw== expires 2026-02-17T06:21:29.516955+0000
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 10 auth: id 2 AQD5B5Rp8yPQHhAA3igz/mVOAybCqpNuLGgTXQ== expires 2026-02-17T06:25:29.516955+0000
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 10 auth: id 3 AQD5B5RpWSvQHhAAJ5lOOfTOsLuWyn1zIyyIrA== expires 2026-02-17T06:29:29.516955+0000
2026-02-17T06:19:53.786+0000 7fe1dabf26c0 1 -- 10.20.193.106:0/3812633100 --> [v2:10.20.193.106:3300/0,v1:10.20.193.106:6789/0] -- log(1 entries from seq 6 at 2026-02-17T06:19:53.400655+0000) -- 0x559c4ac57180 con 0x559c45311400
2026-02-17T06:19:53.901+0000 7fe0ffd0c6c0 0 [prometheus DEBUG prometheus] Port 7789 still in use, waiting...
2026-02-17T06:19:54.223+0000 7fe1d9bf06c0 10 mgr tick tick
Updated by Nitzan Mordechai about 1 month ago
/a/nmordech-2026-02-17_05:44:40-rados:mgr-wip-rocky10-branch-of-the-day-2026-02-13-1771001916-distro-default-trial/51650
Updated by Abhishek Desai about 1 month ago
Tried to reproduce the issue, but is not reproducible with rocky10 build.
[root@ceph-node-00 ~]# journalctl -f | grep security-config
Feb 17 17:59:28 ceph-node-00 ceph-mon[2042]: mon.ceph-node-00@0(leader) e1 handle_command mon_command({"prefix": "orch get-security-config"} v 0)
Feb 17 17:59:28 ceph-node-00 ceph-mon[2042]: log_channel(audit) log [INF] : from='mgr.14354 192.168.100.100:0/3131758036' entity='mgr.ceph-node-00.opzaqd' cmd={"prefix": "orch get-security-config"} : dispatch
Feb 17 17:59:29 ceph-node-00 ceph-mgr[2261]: log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch
Feb 17 17:59:29 ceph-node-00 ceph-mon[2042]: from='mgr.14354 192.168.100.100:0/3131758036' entity='mgr.ceph-node-00.opzaqd' cmd={"prefix": "orch get-security-config"} : dispatch
Feb 17 17:59:30 ceph-node-00 ceph-mon[2042]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch
Feb 17 18:01:59 ceph-node-00 ceph-mon[2042]: mon.ceph-node-00@0(leader) e1 handle_command mon_command({"prefix": "orch get-security-config"} v 0)
Feb 17 18:01:59 ceph-node-00 ceph-mon[2042]: log_channel(audit) log [INF] : from='mgr.14354 192.168.100.100:0/3131758036' entity='mgr.ceph-node-00.opzaqd' cmd={"prefix": "orch get-security-config"} : dispatch
Feb 17 18:01:59 ceph-node-00 ceph-mgr[2261]: log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch
Feb 17 18:02:00 ceph-node-00 ceph-mon[2042]: from='mgr.14354 192.168.100.100:0/3131758036' entity='mgr.ceph-node-00.opzaqd' cmd={"prefix": "orch get-security-config"} : dispatch
Feb 17 18:02:04 ceph-node-00 ceph-mon[2042]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch get-security-config"}]: dispatch
rocky10 image used : quay.ceph.io/ceph-ci/ceph:wip-rocky10-branch-of-the-day-2026-02-09-1770676549-rockylinux-10
Are there any specific scenario I need check for ?
Updated by Nitzan Mordechai about 1 month ago
@Abhishek Desai can you please check with: wip-rocky10-branch-of-the-day-2026-02-13-1771001916, i think it related to https://github.com/ceph/ceph/pull/67227
Updated by Nitzan Mordechai 30 days ago
- Blocks Bug #73930: ceph-mgr modules rely on deprecated python subinterpreters added
Updated by Nitzan Mordechai 26 days ago
/a/nmordech-2026-02-21_21:31:01-rados:mgr-wip-rocky10-branch-of-the-day-2026-02-21-1771692874-tentacle-distro-default-trial/62994
Updated by Nizamudeen A 25 days ago
@Nitzan Mordechai I opened https://github.com/ceph/ceph/pull/67461 for debugging this. I suspect its the same as https://tracker.ceph.com/issues/74819 because I see the same error here as well. To further debug this, can we include the PR that I opened so it shows more in the log.
Updated by Nitzan Mordechai 25 days ago
@Nizamudeen A i'll add it to the next branch of the day
Updated by Nizamudeen A 24 days ago
thanks Nitzan, the last run revealed the actual error
/metrics: 400 (50 bytes) - Content: Absolute URI not allowed if server is not a proxy.
so I think the new cherrypy is strict in processing the urls where the test url concatenation eventually doesn't get normalized by the wsgiserver (which is the new server i've implemented for cherrypy).
so, r = requests.get(base_uri + url, allow_redirects=False), gets processed with http://prom_url:port//metrics, notice the double slash, which never gets normalized. so I just need to adapt the test to strip out the /.
I'll close the current PR and update my cherrypy PR to fix this.
Updated by Nizamudeen A 24 days ago
- Status changed from New to Fix Under Review
- Assignee set to Nizamudeen A
- Pull request ID set to 67227
Updated by Nitzan Mordechai 24 days ago
@Nizamudeen A thanks ! i'll build a new branch-of-the-day with that pr later on
Updated by Nizamudeen A 21 days ago
okay, i don't see this error anymore. and I saw the test passing in the latest run by fetching metrics
2026-02-26T20:42:42.735 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries
2026-02-26T20:42:42.735 DEBUG:tasks.mgr.mgr_test_case:Found prometheus at http://10.20.193.65:7790/ (daemon y/5452)
2026-02-26T20:42:42.738 INFO:tasks.mgr.test_prometheus:/: 200 (176 bytes)- Content: <!DOCTYPE html>
<html>
<head><title>Ceph Exporter</title></head>
<body>
<h1>Ceph Exporter</h1>
<p><a href='/metrics'>Metrics</a></p>
</body>
</html>
2026-02-26T20:42:42.740 INFO:tasks.mgr.test_prometheus:/metrics: 200 (28301 bytes)- Content:
# HELP ceph_health_status Cluster health status
# TYPE ceph_health_status untyped
ceph_health_status 0.0
# HELP ceph_mon_quorum_status Monitors in quorum
# TYPE ceph_mon_quorum_status gauge
ceph_mon_quorum_status{ceph_daemon="mon.a"} 1.0
ceph_mon_quorum_status{ceph_daemon="mon.b"} 1.0
ceph_mon_quorum_status{ceph_daemon="mon.c"} 1.0
# HELP ceph_fs_metadata FS Metadata
# TYPE ceph_fs_metadata untyped
ceph_fs_metadata{data_pools="4",fs_id="1",metadata_pool="3",name="cephfs"} 1.0
# HELP ceph_mds_me
Updated by Upkeep Bot 17 days ago
- Status changed from Fix Under Review to Pending Backport
- Merge Commit set to 805aa0b1ae57911f2f25d96d5d10a6fe5d583ce3
- Fixed In set to v20.3.0-5760-g805aa0b1ae
- Upkeep Timestamp set to 2026-03-03T07:16:41+00:00
Updated by Upkeep Bot 17 days ago
- Copied to Backport #75283: tentacle: Port 7789 still in use, waiting... added