Bug #54029: orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_orch_cli} test failing - Orchestrator - Ceph

Actions

Copy link

Bug #54029

closed

orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_orch_cli} test failing

Added by Adam King about 4 years ago. Updated 8 months ago.

Status:

Resolved

Priority:

High

Assignee:

Adam King

Category:

Target version:

% Done:

Source:

Backport:

quincy,pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

44965

Tags (freeform):

Merge Commit:

884dc766836ee2b8cb2eedccad9e2e79a50658df

Fixed In:

v17.0.0-10701-g884dc766836

Released In:

v18.2.0~2700

Upkeep Timestamp:

2025-07-14T14:42:49+00:00

Tags:

test-failure

Description

This specific test is failing due to the cpehfs-mirror daemon not appearing up in "ceph orch ps" output within 30 seconds of applying it. Seen this on a few different runs over the last 24 hours, two of which were testing a ceph-volume change that is entirely unrelated.

https://pulpito.ceph.com/adking-2022-01-26_05:20:46-orch:cephadm-wip-adk-testing-2022-01-25-2134-distro-basic-smithi/6641250/
https://pulpito.ceph.com/gabrioux-2022-01-26_10:55:38-rados:cephadm-wip-guits-testing-2022-01-26-0910-distro-default-smithi/6641319/
https://pulpito.ceph.com/gabrioux-2022-01-26_13:07:05-orch:cephadm-wip-guits-testing-2022-01-26-0910-distro-default-smithi/6641423

2022-01-26T07:15:36.658 INFO:tasks.cephfs_test_runner:test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) ... ERROR
2022-01-26T07:15:36.658 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:======================================================================
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 30)
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:Ran 1 test in 63.676s
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:======================================================================
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 30)
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Laura Flores about 4 years ago

/a/benhanokh-2022-01-26_21:12:05-rados-WIP_GBH_NCB_new_alloc_map_A6-distro-basic-smithi/6642205
Description: rados/cephadm/workunits/{agent/on mon_election/classic task/test_orch_cli}

Actions

Copy link

Updated by Neha Ojha about 4 years ago

/a/yuriw-2022-01-27_15:12:15-rados-wip-yuri7-testing-2022-01-26-1547-distro-default-smithi/6644411

Actions

Copy link

Updated by Laura Flores about 4 years ago

/a/yuriw-2022-02-11_22:59:19-rados-wip-yuri4-testing-2022-02-11-0858-distro-default-smithi/6677979

Actions

Copy link

Updated by Adam King about 4 years ago

Priority changed from Normal to High
Pull request ID set to 44965

merged in the now linked PR and all the test_orch_cli tests passed in http://pulpito.front.sepia.ceph.com/adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi/. If no more failures are reported in the next few days I'll mark this resolved

Actions

Copy link

Updated by Adam King about 4 years ago

Status changed from New to Fix Under Review

Actions

Copy link

Updated by Laura Flores about 4 years ago

Another failure, but in Quincy:
/a/yuriw-2022-02-28_21:23:00-rados-wip-yuri-testing-2022-02-28-0823-quincy-distro-default-smithi/6712433

Actions

Copy link

Updated by Adam King about 4 years ago

Status changed from Fix Under Review to Pending Backport
Backport set to quincy

Actions

Copy link

Laura Flores wrote:

Backport PR for quincy: https://github.com/ceph/ceph/pull/45625

merged

Actions

Copy link

#14

Updated by Redouane Kachach Elhicou almost 4 years ago

Backport changed from quincy to quincy,pacific

Actions

Copy link

#15

Updated by Redouane Kachach Elhicou almost 4 years ago

Status changed from Pending Backport to Resolved

Actions

Copy link

#16

Updated by Matan Breizman over 3 years ago

Failing with new timeout as well. Should we open a new tracker?

2022-09-16T05:00:33.816 INFO:tasks.cephfs_test_runner:======================================================================
2022-09-16T05:00:33.817 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-09-16T05:00:33.817 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_b5f1ad94e19b4d646020c0622d3e1dd8df5b673f/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 60)
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_b5f1ad94e19b4d646020c0622d3e1dd8df5b673f/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 60s and 0 retries
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:Ran 1 test in 97.566s
2022-09-16T05:00:33.821 INFO:tasks.cephfs_test_runner:
2022-09-16T05:00:33.821 INFO:tasks.cephfs_test_runner:FAILED (errors=1)

/a/yuriw-2022-09-14_13:16:11-rados-wip-yuri6-testing-2022-09-13-1352-distro-default-smithi/7032334

Actions

Copy link

#17