Bug #72272: test_list_enabled_module: Socket error - Address already in use - Ceph - Ceph

Actions

Copy link

Bug #72272

closed

test_list_enabled_module: Socket error - Address already in use

Added by Connor Fawcett 8 months ago. Updated about 1 month ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Category:

build

Target version:

% Done:

Source:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Tags (freeform):

Merge Commit:

Fixed In:

Released In:

Upkeep Timestamp:

Description

/a/skanta-2025-07-19_23:59:58-rados-wip-bharath5-testing-2025-07-18-0518-distro-default-smithi/8397513

teuthology.log:

2025-07-20T01:03:34.664 INFO:tasks.ceph.mgr.y.smithi189.stderr:Traceback (most recent call last):
2025-07-20T01:03:34.664 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
2025-07-20T01:03:34.911 INFO:tasks.ceph.mgr.y.smithi189.stderr:2025-07-20T01:03:34.907+0000 7f446eb5f640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard' while running on mgr.y: OSError("No socket could be created -- (('::', 8443, 0, 0): [Errno 98] Address already in use)")
2025-07-20T01:03:34.911 INFO:tasks.ceph.mgr.y.smithi189.stderr:2025-07-20T01:03:34.907+0000 7f446eb5f640 -1 dashboard.serve:
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:    self.run()
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib64/python3.9/threading.py", line 917, in run
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:    self._target(*self._args, **self._kwargs)
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/servers.py", line 225, in _start_http_thread
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:    self.httpserver.start()
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib/python3.9/site-packages/cheroot/server.py", line 1844, in start
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:2025-07-20T01:03:34.907+0000 7f446eb5f640 -1 Traceback (most recent call last):
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/share/ceph/mgr/dashboard/module.py", line 633, in serve
2025-07-20T01:03:34.912 INFO:tasks.ceph.mgr.y.smithi189.stderr:    cherrypy.engine.start()
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/wspbus.py", line 283, in start
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:    raise e_info
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/wspbus.py", line 268, in start
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:    self.publish('start')
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/wspbus.py", line 248, in publish
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:    raise exc
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:cherrypy.process.wspbus.ChannelFailures: OSError("No socket could be created -- (('::', 8443, 0, 0): [Errno 98] Address already in use)")
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:    self.prepare()
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:  File "/usr/lib/python3.9/site-packages/cheroot/server.py", line 1799, in prepare
2025-07-20T01:03:34.913 INFO:tasks.ceph.mgr.y.smithi189.stderr:    raise socket.error(msg)
2025-07-20T01:03:34.914 INFO:tasks.ceph.mgr.y.smithi189.stderr:OSError: No socket could be created -- (('::', 8443, 0, 0): [Errno 98] Address already in use)
...
...
2025-07-20T01:04:36.327 INFO:tasks.cephfs_test_runner:======================================================================
2025-07-20T01:04:36.327 INFO:tasks.cephfs_test_runner:error: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
2025-07-20T01:04:36.327 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2025-07-20T01:04:36.327 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_6fb25806e4fb7d7745c85caf53ca220d65a090d9/virtualenv/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:    conn = connection.create_connection(
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_6fb25806e4fb7d7745c85caf53ca220d65a090d9/virtualenv/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:    raise err
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_6fb25806e4fb7d7745c85caf53ca220d65a090d9/virtualenv/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:    sock.connect(sa)
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_6fb25806e4fb7d7745c85caf53ca220d65a090d9/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 590, in connect
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:    self._internal_connect(address)
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_6fb25806e4fb7d7745c85caf53ca220d65a090d9/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
2025-07-20T01:04:36.328 INFO:tasks.cephfs_test_runner:    raise _SocketError(err, strerror(err))
2025-07-20T01:04:36.329 INFO:tasks.cephfs_test_runner:ConnectionRefusedError: [Errno 111] Connection refused

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Connor Fawcett 8 months ago

Related to Bug #70669: ERROR: test_list_enabled_module: cephfs resource temporarily unavailable added

Actions

Copy link

Updated by Afreen Misbah 8 months ago

Looks like checking https://pulpito.ceph.com/skanta-2025-07-19_23:59:58-rados-wip-bharath5-testing-2025-07-18-0518-distro-default-smithi/8397502/

Actions

Copy link

Updated by Afreen Misbah 8 months ago

2025-07-20T00:53:48.577 INFO:tasks.ceph.mgr.y.smithi196.stderr:2025-07-20T00:53:48.574+0000 7f44bd43a640 -1 dashboard.serve:
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:2025-07-20T00:53:48.574+0000 7f44bd43a640 -1 Traceback (most recent call last):
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:  File "/usr/share/ceph/mgr/dashboard/module.py", line 364, in serve
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:    cherrypy.engine.start()
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/wspbus.py", line 283, in start
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:    raise e_info
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/wspbus.py", line 268, in start
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:    self.publish('start')
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:  File "/usr/lib/python3.9/site-packages/cherrypy/process/wspbus.py", line 248, in publish
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:    raise exc
2025-07-20T00:53:48.578 INFO:tasks.ceph.mgr.y.smithi196.stderr:cherrypy.process.wspbus.ChannelFailures: Timeout('Port 8443 not free on ::.')
2025-07-20T00:53:48.579 INFO:tasks.ceph.mgr.y.smithi196.stderr:
2025-07-20T00:53:49.971 INFO:tasks.ceph.mon.a.smithi007.stderr:2025-07-20T00:53:49.968+0000 7f60929d6640 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'dashboard' has failed: Timeout('Port 8443 not free on ::.') (MGR_MODULE_ERROR)
2025-07-20T00:53:51.389 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:53:51.389 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:53:51.392 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:53:56.393 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:53:56.394 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:53:56.398 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:01.399 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:01.399 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:01.401 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:06.402 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:06.403 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:06.405 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:11.406 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:11.406 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:11.411 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:16.412 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:16.412 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:16.415 WARNING:tasks.mgr.dashboard.test_mgr_module:Retried #1/2: 'wait_until_rest_api_accessible' raised 'Timed out after 30s and 0 retries'
2025-07-20T00:54:16.915 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:16.915 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:16.918 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:21.919 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:21.919 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:21.922 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:26.923 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:26.923 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:26.926 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:31.927 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:31.927 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:31.930 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:36.931 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:36.932 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:36.934 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:41.935 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:41.935 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:41.938 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)...
2025-07-20T00:54:46.938 INFO:tasks.mgr.dashboard.test_mgr_module:Trying to reach the REST API endpoint
2025-07-20T00:54:46.939 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.196:7790/api/mgr/module
2025-07-20T00:54:46.941 WARNING:tasks.mgr.dashboard.test_mgr_module:Retried #2/2: 'wait_until_rest_api_accessible' raised 'Timed out after 30s and 0 retries'
2025-07-20T00:54:47.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 5
2025-07-20T00:54:47.564 INFO:teuthology.orchestra.run.smithi007.stderr:2025-07-20T00:54:47.560+0000 7f02e9cfa640  1  Processor -- start

yes, its the same issue

Actions

Copy link

Updated by Laura Flores 8 months ago

Afreen Misbah wrote in #note-3:

[...]

yes, its the same issue

@Afreen Misbah this link you checked isn't the same one as Connor linked in the description. IIUC you were verifying by checking the same test on the original run? If so that's fine, but I want to point out that this was the issue in question in case that got missed:
https://pulpito.ceph.com/skanta-2025-07-19_23:59:58-rados-wip-bharath5-testing-2025-07-18-0518-distro-default-smithi/8397513/

This one had the slightly different trace.

Actions

Copy link

Updated by Connor Fawcett 7 months ago

/a/skanta-2025-08-24_15:53:17-rados-wip-bharath4-testing-2025-08-24-0454-distro-default-smithi/8460756

Actions

Copy link

Updated by Jonathan Bailey 7 months ago

/a/skanta-2025-08-05_23:48:19-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8427198
/a/skanta-2025-08-05_23:48:19-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8427222
/a/skanta-2025-08-05_10:12:24-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8424725
/a/skanta-2025-08-28_03:20:37-rados-wip-bharath1-testing-2025-08-26-1433-distro-default-smithi/8467896
/a/skanta-2025-08-28_03:20:37-rados-wip-bharath1-testing-2025-08-26-1433-distro-default-smithi/8467904

Actions

Copy link