Bug #72968
closedtempest: setUpClass get_credentials_provider() failing with "Connection reset by peer"
0%
Description
some tempest tests have been failing with strange connection errors during setUpClass, apparently when trying to connect to the keystone instance? the keystone log doesn't show anything interesting
most recently in https://qa-proxy.ceph.com/teuthology/cbodley-2025-09-10_16:07:52-rgw-wip-72049-distro-default-gibba/8491738/teuthology.log with 3 failures:
==============================
Failed 3 tests - output below:
==============================
setUpClass (tempest.api.object_storage.test_object_formpost.ObjectFormPostTest)
-------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 206, in setUpClass
raise value.with_traceback(trace)
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 191, in setUpClass
cls.setup_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/base.py", line 71, in setup_credentials
super(BaseObjectTest, cls).setup_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 454, in setup_credentials
manager = cls.get_client_manager(roles=credentials_type[1:],
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 748, in get_client_manager
cred_provider = cls._get_credentials_provider()
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 723, in _get_credentials_provider
cls._creds_provider = credentials.get_credentials_provider(
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 141, in get_credentials_provider
**get_dynamic_provider_params(identity_version))
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 72, in get_dynamic_provider_params
admin_creds = admin_creds or get_configured_admin_credentials(
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 257, in get_configured_admin_credentials
credentials = get_credentials(fill_in=fill_in,
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 300, in get_credentials
return auth.get_credentials(auth_url,
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 653, in get_credentials
creds = auth_provider.fill_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 122, in fill_credentials
auth_data = self.get_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 148, in get_auth
self.set_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 157, in set_auth
self.cache = self._get_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 312, in _get_auth
token, auth_data = auth_func(**auth_params)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 190, in get_token
body = self.auth(**kwargs)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 143, in auth
resp, body = self.post(self.auth_url, body=body)
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/rest_client.py", line 300, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 165, in request
resp, resp_body = self.raw_request(url, method,
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/rest_client.py", line 641, in raw_request
resp, resp_body = self.http_obj.request(
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/http.py", line 115, in request
r = super(ClosingHttp, self).request(method, url, retries=retry,
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/request.py", line 78, in request
return self.request_encode_body(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
setUpClass (tempest.api.object_storage.test_object_temp_url_negative.ObjectTempUrlNegativeTest)
-----------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 206, in setUpClass
raise value.with_traceback(trace)
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 191, in setUpClass
cls.setup_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/base.py", line 71, in setup_credentials
super(BaseObjectTest, cls).setup_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 454, in setup_credentials
manager = cls.get_client_manager(roles=credentials_type[1:],
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 748, in get_client_manager
cred_provider = cls._get_credentials_provider()
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 723, in _get_credentials_provider
cls._creds_provider = credentials.get_credentials_provider(
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 141, in get_credentials_provider
**get_dynamic_provider_params(identity_version))
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 72, in get_dynamic_provider_params
admin_creds = admin_creds or get_configured_admin_credentials(
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 257, in get_configured_admin_credentials
credentials = get_credentials(fill_in=fill_in,
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 300, in get_credentials
return auth.get_credentials(auth_url,
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 653, in get_credentials
creds = auth_provider.fill_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 122, in fill_credentials
auth_data = self.get_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 148, in get_auth
self.set_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 157, in set_auth
self.cache = self._get_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 312, in _get_auth
token, auth_data = auth_func(**auth_params)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 190, in get_token
body = self.auth(**kwargs)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 143, in auth
resp, body = self.post(self.auth_url, body=body)
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/rest_client.py", line 300, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 165, in request
resp, resp_body = self.raw_request(url, method,
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/rest_client.py", line 641, in raw_request
resp, resp_body = self.http_obj.request(
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/http.py", line 115, in request
r = super(ClosingHttp, self).request(method, url, retries=retry,
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/request.py", line 78, in request
return self.request_encode_body(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
setUpClass (tempest.api.object_storage.test_container_staticweb.StaticWebTest)
------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 206, in setUpClass
raise value.with_traceback(trace)
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 191, in setUpClass
cls.setup_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/base.py", line 71, in setup_credentials
super(BaseObjectTest, cls).setup_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 454, in setup_credentials
manager = cls.get_client_manager(roles=credentials_type[1:],
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 748, in get_client_manager
cred_provider = cls._get_credentials_provider()
File "/home/ubuntu/cephtest/tempest/tempest/test.py", line 723, in _get_credentials_provider
cls._creds_provider = credentials.get_credentials_provider(
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 141, in get_credentials_provider
**get_dynamic_provider_params(identity_version))
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 72, in get_dynamic_provider_params
admin_creds = admin_creds or get_configured_admin_credentials(
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 257, in get_configured_admin_credentials
credentials = get_credentials(fill_in=fill_in,
File "/home/ubuntu/cephtest/tempest/tempest/common/credentials_factory.py", line 300, in get_credentials
return auth.get_credentials(auth_url,
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 653, in get_credentials
creds = auth_provider.fill_credentials()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 122, in fill_credentials
auth_data = self.get_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 148, in get_auth
self.set_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 157, in set_auth
self.cache = self._get_auth()
File "/home/ubuntu/cephtest/tempest/tempest/lib/auth.py", line 312, in _get_auth
token, auth_data = auth_func(**auth_params)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 190, in get_token
body = self.auth(**kwargs)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 143, in auth
resp, body = self.post(self.auth_url, body=body)
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/rest_client.py", line 300, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "/home/ubuntu/cephtest/tempest/tempest/lib/services/identity/v3/token_client.py", line 165, in request
resp, resp_body = self.raw_request(url, method,
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/rest_client.py", line 641, in raw_request
resp, resp_body = self.http_obj.request(
File "/home/ubuntu/cephtest/tempest/tempest/lib/common/http.py", line 115, in request
r = super(ClosingHttp, self).request(method, url, retries=retry,
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/request.py", line 78, in request
return self.request_encode_body(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
the day before, i saw one failure in https://qa-proxy.ceph.com/teuthology/cbodley-2025-09-06_01:03:01-rgw-wip-72375-distro-default-gibba/8483924/teuthology.log:
{13} setUpClass (tempest.api.object_storage.test_object_temp_url_negative.ObjectTempUrlNegativeTest) [0.000000s] ... FAILED
a rerun failed the same way on a different test in https://qa-proxy.ceph.com/teuthology/cbodley-2025-09-08_16:10:56-rgw-wip-72375-distro-default-gibba/8487836/teuthology.log:
{15} setUpClass (tempest.api.object_storage.test_object_formpost.ObjectFormPostTest) [0.000000s] ... FAILED
Updated by J. Eric Ivancich 6 months ago
- Assignee set to Marcus Watts
Hey @Marcus Watts , are you handling tempest issues? If not, let's figure out who best to assign this to.
Updated by Oguzhan Ozmen 6 months ago
FWIW, I had sent the changes from https://github.com/ceph/ceph/pull/64800 to teuthology last week:
- ceph-ci branch: https://github.com/ceph/ceph-ci/commits/wip-oozmen-64800/
- build: https://shaman.ceph.com/builds/ceph/wip-oozmen-64800/
- just for tempest test on ubuntu and passed: https://pulpito.ceph.com/bcs-ceph-2025-09-10_18:15:01-rgw:tempest-wip-oozmen-64800-distro-default-smithi/
Updated by J. Eric Ivancich 6 months ago
- Assignee changed from Marcus Watts to Tobias Urdin
@Tobias Urdin Would you mind taking a look at this? I understand you have knowledge of keystone.
Updated by Tobias Urdin 6 months ago
unless this happens on every test i think this is just a transient issue for now, there are no errors anywhere which probably indicates that something was slow or that the request got aborted because the keystone service (which in our qa runs under wsgiref.simple_server which is handling one request at a time)
a heads up though, there is a lot of changes in the openstack and python ecosystem that will hit the openstack part of qa in the future, mostly when we update the version we run, such as:
- when issuing pip, that uses setuptools, it no longer will handle wsgi_scripts so the keystone-wsgi-public "binary" will not exist, this is handled transitionally in pbr but will be removed there in the future
- eventlet is removed in the openstack ecosystem, so most services will just provide a WSGI script as a module so we need to migrate, so both these two points combined means we need to migrate to uwsgi or similar
if something breaks really bad just ping me, otherwise i will allocate some time later this year q4 or in q1 next year to do a complete walkthrough of the openstack parts in QA and bump some versions
Updated by Casey Bodley 6 months ago
setUpClass (tempest.api.object_storage.test_object_formpost.ObjectFormPostTest)
ConnectionResetError: [Errno 104] Connection reset by peer
Updated by Casey Bodley 4 months ago
setUpClass (tempest.api.object_storage.test_healthcheck.HealthcheckTest)
setUpClass (tempest.api.object_storage.test_object_formpost_negative.ObjectFormPostNegativeTest)
setUpClass (tempest.api.object_storage.test_container_acl.ObjectTestACLs)
Updated by J. Eric Ivancich 4 months ago
@Casey Bodley Given this is a low priority for Tobias, I'm wondering how best to move forward.
Given that it's urgent, we're hitting it every week during bug scrub. And those of us who review QE runs hit this error every so often and need to know how to process it. So it's consuming others' time.
We could reduce the priority of the bug here. But we also have to consider how well we can support OpenStack.
Updated by Casey Bodley 4 months ago
J. Eric Ivancich wrote in #note-7:
@Casey Bodley Given this is a low priority for Tobias, I'm wondering how best to move forward.
Given that it's urgent, we're hitting it every week during bug scrub. And those of us who review QE runs hit this error every so often and need to know how to process it. So it's consuming others' time.
We could reduce the priority of the bug here. But we also have to consider how well we can support OpenStack.
feel free to change the priority. i just like to keep the known teuthology failures near the top so they're easy to find when reviewing qa results with failures you don't recognize
known failures like this are painful, because it forces us to look extra-closely at rgw/tempest results. i would have missed a real regression in https://github.com/ceph/ceph/pull/63794#issuecomment-3576544792 otherwise
if the failures tracked here were specific to one test case, we could filter that out of the tempest job. but these "Connection reset by peer" failures seem to happen in random test cases
Updated by Tobias Urdin 4 months ago
In the event that this error is caused by Keystone being overloaded due to the development server only being able to serve a single request at a time we might have luck trying to run Keystone under uwsgi and have it handle more requests at the same time. Could we try something like this (untested) and see if that helps? https://github.com/ceph/ceph/pull/66517
I've only looked at the logs linked here so I might be missing something but the error in the topic "get_credentials_provider()" makes me think it's only related to Keystone.
Updated by J. Eric Ivancich 3 months ago
- Status changed from New to In Progress
- Pull request ID set to 66517
Updated by Casey Bodley about 1 month ago
- Related to Bug #74911: rgw/tempest: tempest.api.object_storage.test_object_services.PublicObjectTest added
Updated by Casey Bodley about 1 month ago
- Has duplicate Bug #74911: rgw/tempest: tempest.api.object_storage.test_object_services.PublicObjectTest added
Updated by Casey Bodley 18 days ago
- Is duplicate of Bug #75041: rgw:crypt:barbican fails with ModuleNotFoundError: No module named 'pkg_resources' because of keystone dependency added
Updated by Casey Bodley 18 days ago
- Status changed from In Progress to Duplicate
the fix for https://tracker.ceph.com/issues/75041 includes the same uwsgi changes as https://github.com/ceph/ceph/pull/66517, so i'm hoping this is resolved there. closing this as a duplicate in the meantime