Bug #52109
opentest_cephadm.sh: Failed to connect to grafana.a: Failed to connect to localhost port 3000 after 0 ms: Connection refused
0%
Description
2021-08-08T08:51:36.788 INFO:tasks.workunit.client.0.smithi173.stdout:Aug 08 08:47:54 smithi173 ceph-mgr[45646]: Traceback (most recent call last):
2021-08-08T08:51:36.788 INFO:tasks.workunit.client.0.smithi173.stdout: File "/usr/share/ceph/mgr/dashboard/module.py", line 503, in serve
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: cherrypy.engine.start()
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 283, in start
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: raise e_info
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 268, in start
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout: self.publish('start')
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout: File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 248, in publish
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout: raise exc
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout: cherrypy.process.wspbus.ChannelFailures: Timeout('Port 8443 not free on 127.0.0.1.',)
http://qa-proxy.ceph.com/teuthology/kchai-2021-08-08_08:16:57-rados-wip-kefu-testing-2021-08-08-1031-distro-basic-smithi/6326974/teuthology.log
https://pulpito.ceph.com/sage-2021-08-06_19:04:50-orch-master-distro-basic-smithi/
UPDATED 08/16/2025¶
2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local name=grafana 2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local 'condition=curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\''' 2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local tries=50 2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local num=0 2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval 'curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\''' 2025-07-27T11:10:50.108 INFO:tasks.workunit.client.0.smithi067.stderr:++ curl --insecure https://localhost:3000 2025-07-27T11:10:50.108 INFO:tasks.workunit.client.0.smithi067.stderr:++ grep -q grafana 2025-07-27T11:10:50.111 INFO:tasks.workunit.client.0.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr: Dload Upload Total Spent Left Speed 2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused 2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ num=1 2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -ge 50 ']' 2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ sleep 5 2025-07-27T11:10:55.113 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval 'curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\''' 2025-07-27T11:10:55.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ curl --insecure https://localhost:3000 2025-07-27T11:10:55.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ grep -q grafana 2025-07-27T11:10:55.120 INFO:tasks.workunit.client.0.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current . . . 2025-07-27T11:14:50.631 INFO:tasks.workunit.client.0.smithi067.stderr: Dload Upload Total Spent Left Speed 2025-07-27T11:14:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused 2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:+ num=49 2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 49 -ge 50 ']' 2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:+ sleep 5 2025-07-27T11:14:55.633 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval 'curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\''' 2025-07-27T11:14:55.634 INFO:tasks.workunit.client.0.smithi067.stderr:++ curl --insecure https://localhost:3000 2025-07-27T11:14:55.634 INFO:tasks.workunit.client.0.smithi067.stderr:++ grep -q grafana 2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr: Dload Upload Total Spent Left Speed 2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused 2025-07-27T11:14:55.642 INFO:tasks.workunit.client.0.smithi067.stderr:+ num=50 2025-07-27T11:14:55.642 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 50 -ge 50 ']' 2025-07-27T11:14:55.642 INFO:tasks.workunit.client.0.smithi067.stdout:grafana is not available 2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'grafana is not available' 2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ false 2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ cleanup 2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' true = false ']'
Updated by Daniel Pivonka over 4 years ago
- Status changed from New to Won't Fix
test_cephadm.sh deploys two mgrs on the same host first one from bootstrap and the second with cephadm deploy. both mgrs will want to bind to 127.0.0.1:8443. the second mgrs dashboard module will fail because the first mgrs dashboard module has already binded to 127.0.0.1:8443. this failure is not a fatal error and will not crash the test but can be seen in the second mgrs logs. it does not effect the test though. closing as this doesn't really effect the test and there is no good solution to stop the error.
https://github.com/ceph/ceph/blob/master/qa/workunits/cephadm/test_cephadm.sh#L241
Updated by Laura Flores almost 3 years ago
/a/yuriw-2023-04-26_20:20:05-rados-pacific-release-distro-default-smithi/7255305
Updated by Matan Breizman about 2 years ago
/a/yuriw-2024-02-09_00:15:46-rados-wip-yuri2-testing-2024-02-08-0727-distro-default-smithi/7553385
Updated by Sridhar Seshasayee about 2 years ago
/a/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587370
Updated by Laura Flores almost 2 years ago
/a/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620513
Updated by Laura Flores almost 2 years ago
- Has duplicate Bug #65129: Unhandled Dashboard exception related to CherryPy added
Updated by Laura Flores almost 2 years ago
/a/yuriw-2024-04-09_14:58:25-rados-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648958
Updated by Laura Flores almost 2 years ago
/a/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650646
Updated by Matan Breizman almost 2 years ago
/a/yuriw-2024-04-16_23:25:35-rados-wip-yuriw-testing-20240416.150233-distro-default-smithi/7659292
Updated by Laura Flores almost 2 years ago
/a/yuriw-2024-04-11_17:03:54-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7652453
Updated by Naveen Naidu over 1 year ago
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918907
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7919034
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918959
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918790
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918912
Updated by Shraddha Agrawal over 1 year ago
/a/skanta-2024-12-12_03:27:32-rados-wip-bharath11-testing-2024-12-11-1511-distro-default-smithi/8031486
Updated by Jaya Prakash about 1 year ago
/a/akupczyk-2025-03-07_12:33:54-rados-aclamk-testing-nauvoo-2025-03-04-1719-distro-default-smithi/8174750
Updated by Aishwarya Mathuria 10 months ago
/a/yuriw-2025-05-12_19:14:14-rados-wip-yuri10-testing-2025-05-12-0753-distro-default-smithi/8281631
Updated by Jaya Prakash 10 months ago
/a/akupczyk-2025-05-22_09:41:31-rados-aclamk-testing-phoebe-2025-05-21-1802-distro-default-smithi
2 jobs: ['8293886', '8294015']
Updated by Aishwarya Mathuria 9 months ago
/a/skanta-2025-06-07_23:26:52-rados-wip-bharath5-testing-2025-06-02-2047-distro-default-smithi/8313564
Updated by Shraddha Agrawal 9 months ago
/a/yuriw-2025-05-20_14:56:40-rados-wip-yuri3-testing-2025-05-12-0801-tentacle-distro-default-smithi/8290617
Updated by Shraddha Agrawal 9 months ago
/a/skanta-2025-06-18_11:22:46-rados-wip-yuri-testing-2025-06-12-0731-distro-default-smithi/8334510
Updated by Sridhar Seshasayee 9 months ago
/a/skanta-2025-06-17_15:42:09-rados-wip-bharath13-testing-2025-06-17-1621-distro-default-smithi/8332763
Updated by Kamoltat (Junior) Sirivadhna 9 months ago
/a/skanta-2025-06-07_04:15:58-rados-wip-bharath8-testing-2025-06-02-1508-distro-default-smithi/8312617
Updated by Sridhar Seshasayee 9 months ago
/a/skanta-2025-06-16_03:59:33-rados-wip-bharath10-testing-2025-06-15-0841-distro-default-smithi/8330484
Updated by Shraddha Agrawal 9 months ago
/a/yuriw-2025-06-19_19:29:56-rados-wip-yuri11-testing-2025-06-19-0935-distro-default-smithi/8337249
Updated by Shraddha Agrawal 9 months ago
/a/skanta-2025-06-25_05:53:02-rados-wip-bharath4-testing-2025-06-24-0841-distro-default-smithi/8347691
Updated by Shraddha Agrawal 9 months ago
/a/skanta-2025-06-26_23:33:10-rados-wip-bharath6-testing-2025-06-26-1840-tentacle-distro-default-smithi/8352357
Updated by Lee Sanders 9 months ago
/a/ljsanders-2025-07-01_17:21:46-rados-main-distro-default-smithi/8364091
Updated by Shraddha Agrawal 9 months ago
/a/skanta-2025-07-04_23:32:34-rados-wip-bharath13-testing-2025-07-04-0559-distro-default-smithi/8370611
Updated by Shraddha Agrawal 9 months ago
/a/skanta-2025-07-03_10:29:59-rados-wip-bharath5-testing-2025-06-30-2106-distro-default-smithi/8368520
Updated by Shraddha Agrawal 8 months ago
/a/yuriw-2025-07-10_01:00:46-rados-wip-yuri-testing-2025-07-09-1458-tentacle-distro-default-smithi/8379307
Updated by Shraddha Agrawal 8 months ago
/a/skanta-2025-07-13_23:08:24-rados-wip-bharath4-testing-2025-07-13-0539-distro-default-smithi/8384534
Updated by Shraddha Agrawal 8 months ago
/a/skanta-2025-07-26_22:27:26-rados-wip-bharath7-testing-2025-07-26-0611-tentacle-distro-default-smithi/8409767
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
/a/yuriw-2025-07-28_18:11:33-rados-wip-yuri2-testing-2025-07-24-0816-tentacle-distro-default-smithi/8412121
Updated by Kamoltat (Junior) Sirivadhna 7 months ago · Edited
Think the real issue is that there is a failure in test_cephadm.sh
teuthology-2025-08-10_20:00:42-rados-main-distro-default-smithi/8434834/teuthology.log
2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused 2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:+ num=50 2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:+ '[' 50 -ge 50 ']' 2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:+ echo 'grafana is not available' 2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stdout:grafana is not available 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ false 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ cleanup 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ '[' true = false ']' 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ dump_all_logs 00000000-0000-0000-0000-0000deadbeef 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ local fsid=00000000-0000-0000-0000-0000deadbeef 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:++ jq -r '.[] | select(.fsid == "00000000-0000-0000-0000-0000deadbeef").name' 2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:++ sudo /usr/sbin/cephadm --image quay.ceph.io/ceph-ci/ceph:main ls 2025-08-10T21:26:10.064 INFO:tasks.workunit.client.0.smithi131.stderr:+ local 'names=mon.a 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:mgr.x 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:crash.smithi131 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:mon.b 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:mgr.y 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:osd.0 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:osd.1 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:node-exporter.a 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:prometheus.a 2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:grafana.a' 2025-08-10T21:26:10.066 INFO:tasks.workunit.client.0.smithi131.stdout:dumping logs for daemons: mon.a mgr.x crash.smithi131 mon.b mgr.y osd.0 osd.1 node-exporter.a prometheus.a grafana.a
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
- Assignee changed from Daniel Pivonka to Adam King
I'm changing the title of the bug assigning it to Adam King
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
- Subject changed from test_cephadm.sh: Timeout('Port 8443 not free on 127.0.0.1.',) to test_cephadm.sh: Failed to connect to grafana.a: Failed to connect to localhost port 3000 after 0 ms: Connection refused
- Description updated (diff)
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
The failure is here:
qa/workunits/cephadm/test_cephadm.sh
function is_available()
{
local name="$1"
local condition="$2"
local tries="$3"
local num=0
while ! eval "$condition"; do
num=$(($num + 1))
if [ "$num" -ge $tries ]; then
echo "$name is not available"
false
fi
sleep 5
done
echo "$name is available"
true
}
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
- Status changed from Won't Fix to New
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
/a/teuthology-2025-08-10_20:00:42-rados-main-distro-default-smithi/8434956/
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
- Description updated (diff)
Updated by Lee Sanders 4 months ago · Edited
/a/skanta-2025-11-01_01:12:28-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8578588
/a/skanta-2025-11-01_01:12:28-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8578589
/a/skanta-2025-11-01_01:12:28-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8578608
/a/skanta-2025-10-31_12:25:51-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8577796
/a/skanta-2025-10-31_12:25:51-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8577805
/a/skanta-2025-10-31_12:25:51-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8577944
I think we need to get some focus on the issue, we are seeing this regularly @Kamoltat (Junior) Sirivadhna @Adam King
Updated by Aishwarya Mathuria 4 months ago · Edited
/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605410
/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605281
/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605532
Updated by Sridhar Seshasayee 4 months ago
/a/skanta-2025-11-13_10:26:04-rados-wip-bharath3-testing-2025-11-12-2038-distro-default-smithi/
[8601362, 8601386]
Updated by Kamoltat (Junior) Sirivadhna 4 months ago
/a/skanta-2025-11-01_02:37:10-rados-wip-bharath4-testing-2025-10-31-1459-distro-default-smithi/
8578622
Updated by Kamoltat (Junior) Sirivadhna 4 months ago
@Lee Sanders I agree, @Adam King I assume this test is probably not important for the orchestrator since there has not been much attention to it. Should we consider moving it out or supress it?