Project

General

Profile

Actions
Copy link

Bug #62221

closed

Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)

Added by Venky Shankar over 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jos Collin
Category:
Correctness/Safety
Target version:
Ceph - v20.0.0
% Done:

0%

Source:
Community (dev)
Backport:
quincy,reef,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
cephfs-mirror
Labels (FS):
Pull request ID:
57987
Tags (freeform):
backport_processed
Merge Commit:
f9d1924036df8dbbcc08c5b4fbf6af2afbcb6af1
Fixed In:
v19.3.0-4007-gf9d1924036
Released In:
v20.2.0~2286
Upkeep Timestamp:
2025-11-01T01:34:28+00:00

Description

/a/yuriw-2023-07-26_14:34:38-fs-reef-release-distro-default-smithi/7353194

2023-07-26T23:01:19.423 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/run.py", line 161, in wait
2023-07-26T23:01:19.424 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2023-07-26T23:01:19.424 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/run.py", line 181, in _raise_for_status
2023-07-26T23:01:19.424 INFO:tasks.cephfs_test_runner:    raise CommandFailedError(
2023-07-26T23:01:19.424 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed (mirror status for fs: cephfs) on smithi053 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@2'

The test run the mirror daemon with valgrind. Seems like the daemon didn't start at all. Valgrind report is at: ./remote/smithi053/log/valgrind/cephfs-mirror-client.mirror.log.gz

Related issues 4 (0 open — 4 closed)

Related to CephFS - Bug #64752: cephfs-mirror: valgrind report leaksResolvedJos Collin

Actions
Copied to CephFS - Backport #67390: squid: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)ResolvedJos CollinActions
Copied to CephFS - Backport #67391: quincy: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)ResolvedJos CollinActions
Copied to CephFS - Backport #67392: reef: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)ResolvedJos CollinActions
Actions
Copy link
 #2

Updated by Venky Shankar over 2 years ago

  • Assignee set to Jos Collin

Jos, please take this one.

Actions
Copy link
 #3

Updated by Jos Collin over 2 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #8

Updated by Patrick Donnelly about 2 years ago

/teuthology/yuriw-2024-03-12_14:59:27-fs-wip-yuri11-testing-2024-03-11-0838-reef-distro-default-smithi/7593782/teuthology.log

Actions
Copy link
 #9

Updated by Rishabh Dave almost 2 years ago

Patrick, following is the traceback from the job you reported in the previous comment (/teuthology/yuriw-2024-03-12_14:59:27-fs-wip-yuri11-testing-2024-03-11-0838-reef-distro-default-smithi/7593782/teuthology.log). Copying the traceback here from the log -

2024-03-12T17:59:40.399 INFO:tasks.cephfs_test_runner:======================================================================
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:ERROR: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_49e4a6794d5502bd66908faf714fc54896c51712/qa/tasks/cephfs/test_mirroring.py", line 459, in test_add_ancestor_and_child_directory
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    self.enable_mirroring(self.primary_fs_name, self.primary_fs_id)
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_49e4a6794d5502bd66908faf714fc54896c51712/qa/tasks/cephfs/test_mirroring.py", line 47, in enable_mirroring
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    res = self.mirror_daemon_command(f'counter dump for fs: {fs_name}', 'counter', 'dump')
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_49e4a6794d5502bd66908faf714fc54896c51712/qa/tasks/cephfs/test_mirroring.py", line 283, in mirror_daemon_command
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    p = self.mount_a.client_remote.run(args=
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    r.wait()
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status
2024-03-12T17:59:40.401 INFO:tasks.cephfs_test_runner:    raise CommandFailedError(
2024-03-12T17:59:40.401 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed (counter dump for fs: cephfs) on smithi159 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok counter dump'
2024-03-12T17:59:40.401 INFO:tasks.cephfs_test_runner:

The failing command on your job is ceph counter dump but the failing command on this ticket is ceph fs mirror status. Is the same issue underlying both the failing commands? I am asking because I too found the exact same job failure as you - https://pulpito.ceph.com/yuriw-2024-03-14_15:28:28-fs-wip-yuri4-testing-2024-03-13-0733-reef-distro-default-smithi/7600370

If the underlying issues are different than it's better to open a new ticket for it. I've talked with Jos about this. He'll dig deeper to find it out.

Actions
Copy link
 #10

Updated by Patrick Donnelly almost 2 years ago

Rishabh Dave wrote:

Patrick, following is the traceback from the job you reported in the previous comment (/teuthology/yuriw-2024-03-12_14:59:27-fs-wip-yuri11-testing-2024-03-11-0838-reef-distro-default-smithi/7593782/teuthology.log). Copying the traceback here from the log -

[...]

The failing command on your job is ceph counter dump but the failing command on this ticket is ceph fs mirror status. Is the same issue underlying both the failing commands? I am asking because I too found the exact same job failure as you - https://pulpito.ceph.com/yuriw-2024-03-14_15:28:28-fs-wip-yuri4-testing-2024-03-13-0733-reef-distro-default-smithi/7600370

If the underlying issues are different than it's better to open a new ticket for it. I've talked with Jos about this. He'll dig deeper to find it out.

It may be different as you observed. I didn't look closely.

Actions
Copy link
 #11

Updated by Venky Shankar almost 2 years ago

Rishabh Dave wrote:

Patrick, following is the traceback from the job you reported in the previous comment (/teuthology/yuriw-2024-03-12_14:59:27-fs-wip-yuri11-testing-2024-03-11-0838-reef-distro-default-smithi/7593782/teuthology.log). Copying the traceback here from the log -

[...]

The failing command on your job is ceph counter dump but the failing command on this ticket is ceph fs mirror status. Is the same issue underlying both the failing commands? I am asking because I too found the exact same job failure as you - https://pulpito.ceph.com/yuriw-2024-03-14_15:28:28-fs-wip-yuri4-testing-2024-03-13-0733-reef-distro-default-smithi/7600370

If the underlying issues are different than it's better to open a new ticket for it. I've talked with Jos about this. He'll dig deeper to find it out.

The failed test that Patrick linked is a valgrind run, so it's likely the same underlying issue even if the failure backtrace is different. Jos, please RCA asap.

Actions
Copy link
 #12

Updated by Patrick Donnelly almost 2 years ago

  • Target version changed from v19.0.0 to v20.0.0
Actions
Copy link
 #13

Updated by Jos Collin almost 2 years ago ยท Edited 

2024-03-12T17:59:40.399 INFO:tasks.cephfs_test_runner:======================================================================
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:ERROR: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_49e4a6794d5502bd66908faf714fc54896c51712/qa/tasks/cephfs/test_mirroring.py", line 459, in test_add_ancestor_and_child_directory
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    self.enable_mirroring(self.primary_fs_name, self.primary_fs_id)
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_49e4a6794d5502bd66908faf714fc54896c51712/qa/tasks/cephfs/test_mirroring.py", line 47, in enable_mirroring
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    res = self.mirror_daemon_command(f'counter dump for fs: {fs_name}', 'counter', 'dump')
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_49e4a6794d5502bd66908faf714fc54896c51712/qa/tasks/cephfs/test_mirroring.py", line 283, in mirror_daemon_command
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    p = self.mount_a.client_remote.run(args=
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    r.wait()
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2024-03-12T17:59:40.400 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status
2024-03-12T17:59:40.401 INFO:tasks.cephfs_test_runner:    raise CommandFailedError(
2024-03-12T17:59:40.401 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed (counter dump for fs: cephfs) on smithi159 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok counter dump'
2024-03-12T17:59:40.401 INFO:tasks.cephfs_test_runner:

test_add_ancestor_and_child_directory - first test failed with self.enable_mirroring failure. So the mirror daemon failed to start.
The logs are having the issue mentioned in https://tracker.ceph.com/issues/64752 too. So it's probably the mirror daemon failed to start because of the valgrind CommandFailedError as there's an --show-reachable=yes option.

Actions
Copy link
 #14

Updated by Jos Collin almost 2 years ago

  • Related to Bug #64752: cephfs-mirror: valgrind report leaks added
Actions
Copy link
 #15

Updated by Jos Collin almost 2 years ago

  • Pull request ID set to 57987
Actions
Copy link
 #16

Updated by Jos Collin almost 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Source set to Community (dev)
Actions
Copy link
 #21

Updated by Venky Shankar over 1 year ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #22

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #67390: squid: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) added
Actions
Copy link
 #23

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #67391: quincy: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) added
Actions
Copy link
 #24

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #67392: reef: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) added
Actions
Copy link
 #25

Updated by Upkeep Bot over 1 year ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #26

Updated by Venky Shankar over 1 year ago

  • Backport changed from reef,quincy,pacific to quincy,reef,squid
Actions
Copy link
 #28

Updated by Jos Collin about 1 year ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #29

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to f9d1924036df8dbbcc08c5b4fbf6af2afbcb6af1
  • Fixed In set to v19.3.0-4007-gf9d1924036d
  • Upkeep Timestamp set to 2025-07-12T03:36:02+00:00
Actions
Copy link
 #30

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-4007-gf9d1924036d to v19.3.0-4007-gf9d1924036
  • Upkeep Timestamp changed from 2025-07-12T03:36:02+00:00 to 2025-07-14T23:40:05+00:00
Actions
Copy link
 #31

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~2286
  • Upkeep Timestamp changed from 2025-07-14T23:40:05+00:00 to 2025-11-01T01:34:28+00:00
Actions
Copy link

Also available in: Atom PDF