Project

General

Profile

Actions

Bug #54264

open

Exception type raised when an OrchResult cannot pickle the object

Added by Michael Fritch about 4 years ago. Updated about 4 years ago.

Status:
New
Priority:
Normal
Category:
orchestrator
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

When an OrchResult raises an exception, but fails to pickle:

2022-02-11T15:54:31.914-0700 7f8ed2194640 -1 log_channel(cephadm) log [ERR] : MDS service id cannot start with a numeric digit
Traceback (most recent call last):
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 125, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/home/ceph/src/pybind/mgr/cephadm/module.py", line 2504, in apply
    results.append(self._apply(spec))
  File "/home/ceph/src/pybind/mgr/cephadm/module.py", line 2377, in _apply
    return self._apply_service_spec(cast(ServiceSpec, spec))
  File "/home/ceph/src/pybind/mgr/cephadm/module.py", line 2476, in _apply_service_spec
    HostAssignment(
  File "/home/ceph/src/pybind/mgr/cephadm/schedule.py", line 174, in validate
    self.spec.validate()
  File "/home/ceph/src/python-common/ceph/deployment/service_spec.py", line 684, in validate
    raise SpecValidationError('MDS service id cannot start with a numeric digit')
ceph.deployment.hostspec.SpecValidationError: MDS service id cannot start with a numeric digit
2022-02-11T15:54:31.914-0700 7f8ed2194640  0 [cephadm ERROR orchestrator._interface] failed to pickle MDS service id cannot start with a numeric digit
2022-02-11T15:54:31.918-0700 7f8ed2194640 -1 log_channel(cephadm) log [ERR] : failed to pickle MDS service id cannot start with a numeric digit

it will re-raise with a generic type Exception and break the cli:

# ceph orch apply mds 1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2022-02-11T16:14:37.845-0700 7fa052431640 -1 WARNING: all dangerous and experimental features are enabled.
2022-02-11T16:14:37.857-0700 7fa0511cf640 -1 WARNING: all dangerous and experimental features are enabled.
Error EINVAL: Traceback (most recent call last):
  File "/home/ceph/src/pybind/mgr/mgr_module.py", line 1701, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 171, in handle_command
    return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/home/ceph/src/pybind/mgr/mgr_module.py", line 433, in call
    return self.func(mgr, **kwargs)
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 107, in <lambda>
    wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # noqa: E731
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 96, in wrapper
    return func(*args, **kwargs)
  File "/home/ceph/src/pybind/mgr/orchestrator/module.py", line 1077, in _apply_mds
    return self._apply_misc([spec], dry_run, format, no_overwrite)
  File "/home/ceph/src/pybind/mgr/orchestrator/module.py", line 1046, in _apply_misc
    raise_if_exception(completion)
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 228, in raise_if_exception
    raise e
Exception: MDS service id cannot start with a numeric digit

as seen in this PR: https://github.com/ceph/ceph/pull/44928#discussion_r802152837

Actions #1

Updated by Michael Fritch about 4 years ago

  • Pull request ID set to 44995

still wip (not sure if this is the best way to hanlde:
https://github.com/ceph/ceph/pull/44995

but the origin of a traceback from a failure to pickle is a bit more clear:

2022-02-11T16:14:40.805-0700 7f08b01ab640 -1 log_channel(cephadm) log [ERR] : MDS service id cannot start with a numeric digit
Traceback (most recent call last):
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 125, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/home/ceph/src/pybind/mgr/cephadm/module.py", line 2504, in apply
    results.append(self._apply(spec))
  File "/home/ceph/src/pybind/mgr/cephadm/module.py", line 2377, in _apply
    return self._apply_service_spec(cast(ServiceSpec, spec))
  File "/home/ceph/src/pybind/mgr/cephadm/module.py", line 2476, in _apply_service_spec
    HostAssignment(
  File "/home/ceph/src/pybind/mgr/cephadm/schedule.py", line 174, in validate
    self.spec.validate()
  File "/home/ceph/src/python-common/ceph/deployment/service_spec.py", line 684, in validate
    raise SpecValidationError('MDS service id cannot start with a numeric digit')
ceph.deployment.hostspec.SpecValidationError: MDS service id cannot start with a numeric digit
2022-02-11T16:14:40.805-0700 7f08b01ab640  0 [cephadm ERROR orchestrator._interface] failed to pickle MDS service id cannot start with a numeric digit
2022-02-11T16:14:40.805-0700 7f08b01ab640 -1 log_channel(cephadm) log [ERR] : failed to pickle MDS service id cannot start with a numeric digit
2022-02-11T16:14:40.805-0700 7f08b01ab640 -1 mgr handle_command module 'orchestrator' command handler threw exception: MDS service id cannot start with a numeric digit
2022-02-11T16:14:40.809-0700 7f08b01ab640 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
  File "/home/ceph/src/pybind/mgr/mgr_module.py", line 1701, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 171, in handle_command
    return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/home/ceph/src/pybind/mgr/mgr_module.py", line 433, in call
    return self.func(mgr, **kwargs)
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 107, in <lambda>
    wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # noqa: E731
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 96, in wrapper
    return func(*args, **kwargs)
  File "/home/ceph/src/pybind/mgr/orchestrator/module.py", line 1077, in _apply_mds
    return self._apply_misc([spec], dry_run, format, no_overwrite)
  File "/home/ceph/src/pybind/mgr/orchestrator/module.py", line 1046, in _apply_misc
    raise_if_exception(completion)
  File "/home/ceph/src/pybind/mgr/orchestrator/_interface.py", line 228, in raise_if_exception
    raise e
Exception: MDS service id cannot start with a numeric digit

Actions

Also available in: Atom PDF