Project

General

Profile

Actions
Copy link

Bug #52652

closed

ERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)

Added by Neha Ojha over 4 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Laura Flores
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

2021-09-17T00:55:57.376 INFO:tasks.cephfs_test_runner:test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest) ... ERROR
2021-09-17T00:55:57.376 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.376 INFO:tasks.cephfs_test_runner:======================================================================
2021-09-17T00:55:57.377 INFO:tasks.cephfs_test_runner:ERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)
2021-09-17T00:55:57.377 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-09-17T00:55:57.377 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/mgr/test_module_selftest.py", line 204, in test_module_commands
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner:    self.wait_for_health_clear(timeout=30)
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 169, in wait_for_health_clear
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner:    self.wait_until_true(is_clear, timeout)
2021-09-17T00:55:57.379 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 206, in wait_until_true
2021-09-17T00:55:57.379 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-09-17T00:55:57.379 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:Ran 6 tests in 439.008s
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.381 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2021-09-17T00:55:57.381 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.381 INFO:tasks.cephfs_test_runner:======================================================================
2021-09-17T00:55:57.382 INFO:tasks.cephfs_test_runner:ERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)
2021-09-17T00:55:57.382 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-09-17T00:55:57.382 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/mgr/test_module_selftest.py", line 204, in test_module_commands
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner:    self.wait_for_health_clear(timeout=30)
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 169, in wait_for_health_clear
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner:    self.wait_until_true(is_clear, timeout)
2021-09-17T00:55:57.384 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 206, in wait_until_true
2021-09-17T00:55:57.384 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-09-17T00:55:57.384 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries

/a/yuriw-2021-09-16_18:23:18-rados-wip-yuri2-testing-2021-09-16-0923-distro-basic-smithi/6393748

Last related ticket https://tracker.ceph.com/issues/51743.

Actions
Copy link
 #1

Updated by Neha Ojha over 4 years ago

  • Assignee set to Laura Flores
Actions
Copy link
 #2

Updated by Laura Flores over 4 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #4

Updated by Neha Ojha over 4 years ago

/a/yuriw-2021-12-07_16:04:59-rados-wip-yuri5-testing-2021-12-06-1619-distro-default-smithi/6551099

Actions
Copy link
 #5

Updated by Kamoltat (Junior) Sirivadhna about 4 years ago

/a/yuriw-2021-12-21_18:01:07-rados-wip-yuri3-testing-2021-12-21-0749-distro-default-smithi/6576526/

Actions
Copy link
 #6

Updated by Sridhar Seshasayee about 4 years ago

/a/yuriw-2022-01-08_17:57:43-rados-wip-yuri8-testing-2022-01-07-1541-distro-default-smithi/6603271

Actions
Copy link
 #7

Updated by Laura Flores about 4 years ago

This line is normal in the teuthology log, as this particular test invokes a synthetic exception. However...

2021-09-17T00:55:29.138 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'selftest' has failed: Synthetic exception in serve","count":
1},"muted":false}},"mutes":[]}

An extra line shows up in failed tests, indicating that a mgr module has crashed. This line is not invoked by the `test_module_commands` test. I believe this module crash is what's causing this test to fail (not deterministically) since it relies on a clear health message to pass.

2021-09-17T00:55:34.533 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]}

Actions
Copy link
 #8

Updated by Laura Flores almost 4 years ago

/a/yuriw-2022-06-02_20:23:28-rados-wip-yuri-testing-2022-06-02-0810-quincy-distro-default-smithi/6860841

Actions
Copy link
 #9

Updated by Kamoltat (Junior) Sirivadhna over 3 years ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944203/

Actions
Copy link
 #10

Updated by Laura Flores almost 3 years ago

  • Status changed from In Progress to Closed

Haven't been seeing this.

Actions
Copy link
 #11

Updated by Aishwarya Mathuria about 1 year ago ยท Edited

Seeing this in the following failure: /a/skanta-2024-12-05_07:34:09-rados-wip-bharath2-testing-2024-12-04-1214-distro-default-smithi/8021718

2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:test_selftest_command_spam (tasks.mgr.test_module_selftest.TestModuleSelftest) ... ERROR
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:======================================================================
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:ERROR: test_selftest_command_spam (tasks.mgr.test_module_selftest.TestModuleSelftest)
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:    conn = connection.create_connection(
2024-12-05T08:39:52.806 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:    raise err
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:    sock.connect(sa)
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 590, in connect
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:    self._internal_connect(address)
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:    raise _SocketError(err, strerror(err))
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:ConnectionRefusedError: [Errno 111] Connection refused
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred:
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:    httplib_response = self._make_request(
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
2024-12-05T08:39:52.807 INFO:tasks.cephfs_test_runner:    self._validate_conn(conn)
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:    conn.connect()
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:    self.sock = conn = self._new_conn()
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:    raise NewConnectionError(
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff83a1db8e0>: Failed to establish a new connection: [Errno 111] Connection refused
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred:
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:    resp = conn.urlopen(
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:    retries = retries.increment(
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e4eeebda7d988dc20ee63a4f7e7d82e07061f6d5/virtualenv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:    raise MaxRetryError(_pool, url, error or ResponseError(cause))
2024-12-05T08:39:52.808 INFO:tasks.cephfs_test_runner:urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.21.15.176', port=7790): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff83a1db8e0>: Failed to establish a new connection: [Errno 111] Connection refused'))

2024-12-05T08:33:19.153 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'selftest' has failed: Synthetic exception in serve","count":1},"muted":false}},"mutes":[]}
Actions
Copy link

Also available in: Atom PDF