Project

General

Profile

Actions

Bug #52109

open

test_cephadm.sh: Failed to connect to grafana.a: Failed to connect to localhost port 3000 after 0 ms: Connection refused

Added by Sebastian Wagner over 4 years ago. Updated 4 months ago.

Status:
New
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

https://sentry.ceph.com/organizations/ceph/issues/1585/?project=2&query=is%3Aunresolved+%22workunit+test+cephadm%22&statsPeriod=14d

2021-08-08T08:51:36.788 INFO:tasks.workunit.client.0.smithi173.stdout:Aug 08 08:47:54 smithi173 ceph-mgr[45646]: Traceback (most recent call last):
2021-08-08T08:51:36.788 INFO:tasks.workunit.client.0.smithi173.stdout:                                             File "/usr/share/ceph/mgr/dashboard/module.py", line 503, in serve
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout:                                               cherrypy.engine.start()
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout:                                             File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 283, in start
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout:                                               raise e_info
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout:                                             File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 268, in start
2021-08-08T08:51:36.789 INFO:tasks.workunit.client.0.smithi173.stdout:                                               self.publish('start')
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout:                                             File "/lib/python3.6/site-packages/cherrypy/process/wspbus.py", line 248, in publish
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout:                                               raise exc
2021-08-08T08:51:36.790 INFO:tasks.workunit.client.0.smithi173.stdout:                                           cherrypy.process.wspbus.ChannelFailures: Timeout('Port 8443 not free on 127.0.0.1.',)

http://qa-proxy.ceph.com/teuthology/kchai-2021-08-08_08:16:57-rados-wip-kefu-testing-2021-08-08-1031-distro-basic-smithi/6326974/teuthology.log
https://pulpito.ceph.com/sage-2021-08-06_19:04:50-orch-master-distro-basic-smithi/

UPDATED 08/16/2025

2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local name=grafana
2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local 'condition=curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\'''
2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local tries=50
2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ local num=0
2025-07-27T11:10:50.107 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval 'curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\'''
2025-07-27T11:10:50.108 INFO:tasks.workunit.client.0.smithi067.stderr:++ curl --insecure https://localhost:3000
2025-07-27T11:10:50.108 INFO:tasks.workunit.client.0.smithi067.stderr:++ grep -q grafana
2025-07-27T11:10:50.111 INFO:tasks.workunit.client.0.smithi067.stderr:  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:                                 Dload  Upload   Total   Spent    Left  Speed
2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused
2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ num=1
2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -ge 50 ']'
2025-07-27T11:10:50.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ sleep 5
2025-07-27T11:10:55.113 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval 'curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\'''
2025-07-27T11:10:55.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ curl --insecure https://localhost:3000
2025-07-27T11:10:55.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ grep -q grafana
2025-07-27T11:10:55.120 INFO:tasks.workunit.client.0.smithi067.stderr:  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
.
.
.
2025-07-27T11:14:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:                                 Dload  Upload   Total   Spent    Left  Speed
2025-07-27T11:14:50.631 INFO:tasks.workunit.client.0.smithi067.stderr:^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused
2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:+ num=49
2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 49 -ge 50 ']'
2025-07-27T11:14:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:+ sleep 5
2025-07-27T11:14:55.633 INFO:tasks.workunit.client.0.smithi067.stderr:+ eval 'curl --insecure '\''https://localhost:3000'\'' | grep -q '\''grafana'\'''
2025-07-27T11:14:55.634 INFO:tasks.workunit.client.0.smithi067.stderr:++ curl --insecure https://localhost:3000
2025-07-27T11:14:55.634 INFO:tasks.workunit.client.0.smithi067.stderr:++ grep -q grafana
2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:                                 Dload  Upload   Total   Spent    Left  Speed
2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2025-07-27T11:14:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused
2025-07-27T11:14:55.642 INFO:tasks.workunit.client.0.smithi067.stderr:+ num=50
2025-07-27T11:14:55.642 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 50 -ge 50 ']'
2025-07-27T11:14:55.642 INFO:tasks.workunit.client.0.smithi067.stdout:grafana is not available
2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'grafana is not available'
2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ false
2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ cleanup
2025-07-27T11:14:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' true = false ']'

Related issues 1 (0 open1 closed)

Has duplicate Dashboard - Bug #65129: Unhandled Dashboard exception related to CherryPyDuplicate

Actions
Actions #1

Updated by Sebastian Wagner over 4 years ago

  • Assignee set to Daniel Pivonka
Actions #2

Updated by Daniel Pivonka over 4 years ago

  • Status changed from New to Won't Fix

test_cephadm.sh deploys two mgrs on the same host first one from bootstrap and the second with cephadm deploy. both mgrs will want to bind to 127.0.0.1:8443. the second mgrs dashboard module will fail because the first mgrs dashboard module has already binded to 127.0.0.1:8443. this failure is not a fatal error and will not crash the test but can be seen in the second mgrs logs. it does not effect the test though. closing as this doesn't really effect the test and there is no good solution to stop the error.

https://github.com/ceph/ceph/blob/master/qa/workunits/cephadm/test_cephadm.sh#L241

Actions #3

Updated by Laura Flores almost 3 years ago

/a/yuriw-2023-04-26_20:20:05-rados-pacific-release-distro-default-smithi/7255305

Actions #4

Updated by Matan Breizman about 2 years ago

/a/yuriw-2024-02-09_00:15:46-rados-wip-yuri2-testing-2024-02-08-0727-distro-default-smithi/7553385

Actions #5

Updated by Sridhar Seshasayee about 2 years ago

/a/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587370

Actions #6

Updated by Laura Flores almost 2 years ago

/a/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620513

Actions #7

Updated by Laura Flores almost 2 years ago

  • Has duplicate Bug #65129: Unhandled Dashboard exception related to CherryPy added
Actions #8

Updated by Laura Flores almost 2 years ago

/a/yuriw-2024-04-09_14:58:25-rados-wip-yuri4-testing-2024-04-08-1432-distro-default-smithi/7648958

Actions #9

Updated by Laura Flores almost 2 years ago

/a/yuriw-2024-04-10_14:17:51-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650646

Actions #10

Updated by Matan Breizman almost 2 years ago

/a/yuriw-2024-04-16_23:25:35-rados-wip-yuriw-testing-20240416.150233-distro-default-smithi/7659292

Actions #11

Updated by Laura Flores almost 2 years ago

/a/yuriw-2024-04-11_17:03:54-rados-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7652453

Actions #12

Updated by Naveen Naidu over 1 year ago

/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918907
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7919034
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918959
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918790
/a/skanta-2024-09-25_00:14:38-rados-wip-bharath4-testing-2024-09-24-1154-distro-default-smithi/7918912

Actions #13

Updated by Shraddha Agrawal over 1 year ago

/a/skanta-2024-12-12_03:27:32-rados-wip-bharath11-testing-2024-12-11-1511-distro-default-smithi/8031486

Actions #14

Updated by Jaya Prakash about 1 year ago

/a/akupczyk-2025-03-07_12:33:54-rados-aclamk-testing-nauvoo-2025-03-04-1719-distro-default-smithi/8174750

Actions #15

Updated by Aishwarya Mathuria 10 months ago

/a/yuriw-2025-05-12_19:14:14-rados-wip-yuri10-testing-2025-05-12-0753-distro-default-smithi/8281631

Actions #16

Updated by Jaya Prakash 10 months ago

/a/akupczyk-2025-05-22_09:41:31-rados-aclamk-testing-phoebe-2025-05-21-1802-distro-default-smithi
2 jobs: ['8293886', '8294015']

Actions #17

Updated by Aishwarya Mathuria 9 months ago

/a/skanta-2025-06-07_23:26:52-rados-wip-bharath5-testing-2025-06-02-2047-distro-default-smithi/8313564

Actions #18

Updated by Shraddha Agrawal 9 months ago

/a/yuriw-2025-05-20_14:56:40-rados-wip-yuri3-testing-2025-05-12-0801-tentacle-distro-default-smithi/8290617

Actions #19

Updated by Shraddha Agrawal 9 months ago

/a/skanta-2025-06-18_11:22:46-rados-wip-yuri-testing-2025-06-12-0731-distro-default-smithi/8334510

Actions #20

Updated by Sridhar Seshasayee 9 months ago

/a/skanta-2025-06-17_15:42:09-rados-wip-bharath13-testing-2025-06-17-1621-distro-default-smithi/8332763

Actions #21

Updated by Kamoltat (Junior) Sirivadhna 9 months ago

/a/skanta-2025-06-07_04:15:58-rados-wip-bharath8-testing-2025-06-02-1508-distro-default-smithi/8312617

Actions #22

Updated by Sridhar Seshasayee 9 months ago

/a/skanta-2025-06-16_03:59:33-rados-wip-bharath10-testing-2025-06-15-0841-distro-default-smithi/8330484

Actions #23

Updated by Shraddha Agrawal 9 months ago

/a/yuriw-2025-06-19_19:29:56-rados-wip-yuri11-testing-2025-06-19-0935-distro-default-smithi/8337249

Actions #24

Updated by Shraddha Agrawal 9 months ago

/a/skanta-2025-06-25_05:53:02-rados-wip-bharath4-testing-2025-06-24-0841-distro-default-smithi/8347691

Actions #25

Updated by Shraddha Agrawal 9 months ago

/a/skanta-2025-06-26_23:33:10-rados-wip-bharath6-testing-2025-06-26-1840-tentacle-distro-default-smithi/8352357

Actions #26

Updated by Lee Sanders 9 months ago

/a/ljsanders-2025-07-01_17:21:46-rados-main-distro-default-smithi/8364091

Actions #27

Updated by Shraddha Agrawal 9 months ago

/a/skanta-2025-07-04_23:32:34-rados-wip-bharath13-testing-2025-07-04-0559-distro-default-smithi/8370611

Actions #28

Updated by Shraddha Agrawal 9 months ago

/a/skanta-2025-07-03_10:29:59-rados-wip-bharath5-testing-2025-06-30-2106-distro-default-smithi/8368520

Actions #29

Updated by Shraddha Agrawal 8 months ago

/a/yuriw-2025-07-10_01:00:46-rados-wip-yuri-testing-2025-07-09-1458-tentacle-distro-default-smithi/8379307

Actions #30

Updated by Shraddha Agrawal 8 months ago

/a/skanta-2025-07-13_23:08:24-rados-wip-bharath4-testing-2025-07-13-0539-distro-default-smithi/8384534

Actions #31

Updated by Shraddha Agrawal 8 months ago

/a/skanta-2025-07-26_22:27:26-rados-wip-bharath7-testing-2025-07-26-0611-tentacle-distro-default-smithi/8409767

Actions #32

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

/a/yuriw-2025-07-28_18:11:33-rados-wip-yuri2-testing-2025-07-24-0816-tentacle-distro-default-smithi/8412121

Actions #33

Updated by Kamoltat (Junior) Sirivadhna 7 months ago · Edited

Think the real issue is that there is a failure in test_cephadm.sh

teuthology-2025-08-10_20:00:42-rados-main-distro-default-smithi/8434834/teuthology.log

2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused
2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:+ num=50
2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:+ '[' 50 -ge 50 ']'
2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stderr:+ echo 'grafana is not available'
2025-08-10T21:26:04.757 INFO:tasks.workunit.client.0.smithi131.stdout:grafana is not available
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ false
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ cleanup
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ '[' true = false ']'
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ dump_all_logs 00000000-0000-0000-0000-0000deadbeef
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:+ local fsid=00000000-0000-0000-0000-0000deadbeef
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:++ jq -r '.[] | select(.fsid == "00000000-0000-0000-0000-0000deadbeef").name'
2025-08-10T21:26:04.758 INFO:tasks.workunit.client.0.smithi131.stderr:++ sudo /usr/sbin/cephadm --image quay.ceph.io/ceph-ci/ceph:main ls
2025-08-10T21:26:10.064 INFO:tasks.workunit.client.0.smithi131.stderr:+ local 'names=mon.a
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:mgr.x
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:crash.smithi131
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:mon.b
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:mgr.y
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:osd.0
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:osd.1
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:node-exporter.a
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:prometheus.a
2025-08-10T21:26:10.065 INFO:tasks.workunit.client.0.smithi131.stderr:grafana.a'
2025-08-10T21:26:10.066 INFO:tasks.workunit.client.0.smithi131.stdout:dumping logs for daemons:  mon.a mgr.x crash.smithi131 mon.b mgr.y osd.0 osd.1 node-exporter.a prometheus.a grafana.a
Actions #34

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

  • Assignee changed from Daniel Pivonka to Adam King

I'm changing the title of the bug assigning it to Adam King

Actions #35

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

  • Subject changed from test_cephadm.sh: Timeout('Port 8443 not free on 127.0.0.1.',) to test_cephadm.sh: Failed to connect to grafana.a: Failed to connect to localhost port 3000 after 0 ms: Connection refused
  • Description updated (diff)
Actions #36

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

The failure is here:

qa/workunits/cephadm/test_cephadm.sh

function is_available()
{
    local name="$1" 
    local condition="$2" 
    local tries="$3" 

    local num=0
    while ! eval "$condition"; do
        num=$(($num + 1))
        if [ "$num" -ge $tries ]; then
            echo "$name is not available" 
            false
        fi
        sleep 5
    done

    echo "$name is available" 
    true
}

Actions #37

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

  • Status changed from Won't Fix to New
Actions #38

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

pinged Adam King

Actions #39

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

/a/teuthology-2025-08-10_20:00:42-rados-main-distro-default-smithi/8434956/

Actions #40

Updated by Kamoltat (Junior) Sirivadhna 7 months ago

  • Description updated (diff)
Actions #41

Updated by Lee Sanders 4 months ago · Edited

/a/skanta-2025-11-01_01:12:28-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8578588
/a/skanta-2025-11-01_01:12:28-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8578589
/a/skanta-2025-11-01_01:12:28-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8578608
/a/skanta-2025-10-31_12:25:51-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8577796
/a/skanta-2025-10-31_12:25:51-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8577805
/a/skanta-2025-10-31_12:25:51-rados-wip-bharath5-testing-2025-10-31-1454-distro-default-smithi/8577944

I think we need to get some focus on the issue, we are seeing this regularly @Kamoltat (Junior) Sirivadhna @Adam King

Actions #42

Updated by Aishwarya Mathuria 4 months ago · Edited

/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605410
/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605281
/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605532

Actions #43

Updated by Sridhar Seshasayee 4 months ago

/a/skanta-2025-11-13_10:26:04-rados-wip-bharath3-testing-2025-11-12-2038-distro-default-smithi/
[8601362, 8601386]

Actions #44

Updated by Kamoltat (Junior) Sirivadhna 4 months ago

/a/skanta-2025-11-01_02:37:10-rados-wip-bharath4-testing-2025-10-31-1459-distro-default-smithi/
8578622

Actions #45

Updated by Kamoltat (Junior) Sirivadhna 4 months ago

@Lee Sanders I agree, @Adam King I assume this test is probably not important for the orchestrator since there has not been much attention to it. Should we consider moving it out or supress it?

Actions

Also available in: Atom PDF