Bug #72871: rados/thrash-old-clients - [cephadm ERROR orchestrator._interface] Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) - Ceph - Ceph

Actions

Copy link

Bug #72871

closed

rados/thrash-old-clients - [cephadm ERROR orchestrator._interface] Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout)

Added by Lee Sanders 7 months ago. Updated 7 months ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

ceph-deploy

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Tags (freeform):

Merge Commit:

Fixed In:

Released In:

Upkeep Timestamp:

Description

2025-08-19T20:08:06.267 INFO:journalctl@ceph.mon.b.smithi064.stdout:Aug 19 20:08:05 smithi064 ceph-mon³⁵³⁴⁴: 1.0 scrub starts
2025-08-19T20:08:06.268 INFO:journalctl@ceph.mon.b.smithi064.stdout:Aug 19 20:08:05 smithi064 ceph-mon³⁵³⁴⁴: 1.0 scrub ok
2025-08-19T20:08:06.312 INFO:journalctl@ceph.mon.a.smithi043.stdout:Aug 19 20:08:05 smithi043 ceph-mon³²²⁷⁸: 1.0 scrub starts
2025-08-19T20:08:06.312 INFO:journalctl@ceph.mon.a.smithi043.stdout:Aug 19 20:08:05 smithi043 ceph-mon³²²⁷⁸: 1.0 scrub ok
2025-08-19T20:08:06.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Aug 19 20:08:05 smithi169 ceph-mon³⁵⁴⁶⁹: 1.0 scrub starts
2025-08-19T20:08:06.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Aug 19 20:08:05 smithi169 ceph-mon³⁵⁴⁶⁹: 1.0 scrub ok
2025-08-19T20:08:07.813 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: 2025-08-19T20:08:07.476+0000 7f189c8f0640 -1 log_channel(cephadm) log [ERR] : Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout)
2025-08-19T20:08:07.813 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: Traceback (most recent call last):
2025-08-19T20:08:07.814 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/module.py", line 829, in async_timeout_handler
2025-08-19T20:08:07.814 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: yield
2025-08-19T20:08:07.814 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 81, in create_from_spec
2025-08-19T20:08:07.814 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: ret = self.mgr.wait_async(all_hosts())
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/module.py", line 818, in wait_async
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: return self.event_loop.get_result(coro, timeout)
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: return future.result(timeout)
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/lib64/python3.9/concurrent/futures/_base.py", line 448, in result
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: raise TimeoutError()
2025-08-19T20:08:07.815 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: concurrent.futures._base.TimeoutError
2025-08-19T20:08:07.816 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹:
2025-08-19T20:08:07.816 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: During handling of the above exception, another exception occurred:
2025-08-19T20:08:07.816 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹:
2025-08-19T20:08:07.816 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: Traceback (most recent call last):
2025-08-19T20:08:07.816 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
2025-08-19T20:08:07.816 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: return OrchResult(f(*args, **kwargs))
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/module.py", line 3074, in create_osds
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: return self.osd_service.create_from_spec(drive_group)
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 81, in create_from_spec
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: ret = self.mgr.wait_async(all_hosts())
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/lib64/python3.9/contextlib.py", line 137, in exit
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: self.gen.throw(typ, value, traceback)
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: File "/usr/share/ceph/mgr/cephadm/module.py", line 842, in async_timeout_handler
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: raise OrchestratorError(err_str)
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: orchestrator._interface.OrchestratorError: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout)
2025-08-19T20:08:07.817 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Aug 19 20:08:07 smithi043 ceph-40dce292-7d35-11f0-8741-adfe0268badd-mgr-y³²⁵⁰⁹: 2025-08-19T20:08:07.476+0000 7f189c8f0640 -1 mgr.server reply reply (22) Invalid argument Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout)
2025-08-19T20:08:12.067 INFO:journalctl@ceph.mon.a.smithi043.stdout:Aug 19 20:08:11 smithi043 ceph-mon³²²⁷⁸: purged_snaps scrub starts
2025-08-19T20:08:12.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Aug 19 20:08:11 smithi043 ceph-mon³²²⁷⁸: purged_snaps scrub ok