Actions
Bug #70247
openNon-zero exit code 1 from systemctl reset-failed ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Description
Following logs are from job number 8164269:
2025-03-02T12:27:09.613 INFO:teuthology.orchestra.run.smithi017.stdout:Creating mon... 2025-03-02T12:27:10.476 INFO:teuthology.orchestra.run.smithi017.stdout:create mon.a on 2025-03-02T12:27:10.669 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2025-03-02T12:27:10.888 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2025-03-02T12:27:11.065 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target → /etc/systemd/system/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target. 2025-03-02T12:27:11.066 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target → /etc/systemd/system/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target. 2025-03-02T12:27:11.313 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a 2025-03-02T12:27:11.313 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to reset failed state of unit ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service: Unit ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service not loaded. 2025-03-02T12:27:11.486 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target.wants/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service → /etc/systemd/system/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@.service. 2025-03-02T12:27:11.866 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl start ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a 2025-03-02T12:27:11.866 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Job for ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service failed because the control process exited with error code. 2025-03-02T12:27:11.867 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr See "systemctl status ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service" and "journalctl -xeu ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service" for details. 2025-03-02T12:27:11.867 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl start failed for ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a: Failed command: systemctl start ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a: Job for ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service failed because the control process exited with error code. 2025-03-02T12:27:11.867 INFO:teuthology.orchestra.run.smithi017.stderr:See "systemctl status ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service" and "journalctl -xeu ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service" for details. 2025-03-02T12:27:11.867 INFO:teuthology.orchestra.run.smithi017.stderr: 2025-03-02T12:27:11.867 INFO:teuthology.orchestra.run.smithi017.stderr:DaemonStartException: 2025-03-02T12:27:11.867 INFO:teuthology.orchestra.run.smithi017.stdout: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: *************** 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: Cephadm hit an issue during cluster installation. Current cluster files will be deleted automatically. 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: To disable this behaviour you can pass the --no-cleanup-on-failure flag. In case of any previous 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: broken installation, users must use the following command to completely delete the broken cluster: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: > cephadm rm-cluster --force --zap-osds --fsid <fsid> 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: for more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: *************** 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout: 2025-03-02T12:27:11.868 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 47356c0e-f761-11ef-bb88-bd4984dce30f 2025-03-02T12:27:11.922 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 02 12:27:11 smithi017 systemd[1]: Failed to start Ceph mon.a for 47356c0e-f761-11ef-bb88-bd4984dce30f. 2025-03-02T12:27:11.922 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 02 12:27:11 smithi017 systemd[1]: Stopped Ceph mon.a for 47356c0e-f761-11ef-bb88-bd4984dce30f. 2025-03-02T12:27:12.095 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Removed "/etc/systemd/system/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target.wants/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service". 2025-03-02T12:27:12.199 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 5 from systemctl stop ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service 2025-03-02T12:27:12.199 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to stop ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service: Unit ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service not loaded. 2025-03-02T12:27:12.205 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service 2025-03-02T12:27:12.206 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to reset failed state of unit ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service: Unit ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service not loaded. 2025-03-02T12:27:12.212 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl disable ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service 2025-03-02T12:27:12.212 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to disable unit: Unit file ceph-47356c0e-f761-11ef-bb88-bd4984dce30f-init@mon.a.service does not exist. 2025-03-02T12:27:12.373 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Removed "/etc/systemd/system/ceph.target.wants/ceph-47356c0e-f761-11ef-bb88-bd4984dce30f.target". 2025-03-02T12:27:12.542 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2025-03-02T12:27:12.543 INFO:teuthology.orchestra.run.smithi017.stderr:Traceback (most recent call last): 2025-03-02T12:27:12.544 INFO:teuthology.orchestra.run.smithi017.stderr: File "/tmp/tmpq438djx_.cephadm.build/app/__main__.py", line 1206, in deploy_daemon_units 2025-03-02T12:27:12.544 INFO:teuthology.orchestra.run.smithi017.stderr: File "/tmp/tmpq438djx_.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2025-03-02T12:27:12.544 INFO:teuthology.orchestra.run.smithi017.stderr:RuntimeError: Failed command: systemctl start ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a: Job for ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service failed because the control process exited with error code. 2025-03-02T12:27:12.544 INFO:teuthology.orchestra.run.smithi017.stderr:See "systemctl status ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service" and "journalctl -xeu ceph-47356c0e-f761-11ef-bb88-bd4984dce30f@mon.a.service" for details.
My teutholgy HEAD is:
commit 102abe6af4482f75c77e160b1f63b62998fb9295 (HEAD -> main, origin/main, origin/HEAD)
Merge: 6f4706ab 4c25626a
Author: Zack Cerza <zack@redhat.com>
Date: Thu Feb 27 14:09:29 2025 -0700
Merge pull request #2030 from jmundack/add_codeowners
Add CODEOWNERS file
Looks like this has resurfaced again.
The only addition on the cephadm side from me is:
commit 5a261882aa7001d843e5d8b72a95e4c39331df79 (HEAD -> use-libc-for-segfault, ci/wip-mchangir-use-libc-for-segfault-main-debug)
Author: Milind Changire <mchangir@redhat.com>
Date: Sat Mar 1 21:59:28 2025 +0530
cephadm: tweak to help generate core dump in container
Signed-off-by: Milind Changire <mchangir@redhat.com>
diff --git a/src/cephadm/cephadmlib/container_engines.py b/src/cephadm/cephadmlib/container_engines.py
index 9b308fdd417..8c730c2e34b 100644
--- a/src/cephadm/cephadmlib/container_engines.py
+++ b/src/cephadm/cephadmlib/container_engines.py
@@ -79,6 +79,7 @@ class Podman(ContainerEngine):
f'{runtime_dir}/{service_name}-pid',
'--cidfile',
f'{runtime_dir}/{service_name}-cid',
+ '--ulimit=core=unlimited'
]
)
if self.supports_split_cgroups and not ctx.no_cgroups_split:
Updated by Venky Shankar about 1 year ago
- Related to Bug #69953: mds: segmentation faults in recent QA added
Updated by Nitzan Mordechai 6 months ago
- Has duplicate Bug #72874: rados/thrash-old-clients Stuck doing _try_send injecting socket failure and then nothing for 8 hours added
Updated by Nitzan Mordechai 6 months ago
- Has duplicate Bug #72890: rados/thrash-old-clients cluster create and lots of scrubs then timed out added
Actions