[FilesystemBackend] Failure in unpickling a FilesystemBackend instance for a group #6108
Description
Checklist
- [ x] I have verified that the issue exists against the
masterbranch of Celery. - This has already been asked to the discussion group first.
- [x ] I have read the relevant section in the
contribution guide
on reporting bugs. - [x ] I have checked the issues list
for similar or identical bug reports. - I have checked the pull requests list
for existing proposed fixes. - I have checked the commit log
to find out if the bug was already fixed in the master branch. - I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
- I have included the output of
celery -A proj reportin the issue.
(if you are not able to do this, then at least specify the Celery
version affected). - I have verified that the issue exists against the
masterbranch of Celery. - I have included the contents of
pip freezein the issue. - I have included all the versions of all the external dependencies required
to reproduce this bug.
Optional Debugging Information
- I have tried reproducing the issue on more than one Python version
and/or implementation. - I have tried reproducing the issue on more than one message broker and/or
result backend. - I have tried reproducing the issue on more than one version of the message
broker and/or result backend. - I have tried reproducing the issue on more than one operating system.
- I have tried reproducing the issue on more than one workers pool.
- I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled. - I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
- None
Possible Duplicates
- None
Environment & Settings
Celery version:
celery report Output:
software -> celery:4.4.2 (cliffs) kombu:4.6.8 py:3.7.5
billiard:3.6.3.0 py-amqp:2.5.2
platform -> system:Darwin arch:64bit
kernel version:18.7.0 imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled
Steps to Reproduce
Required Dependencies
- Minimal Python Version: N/A or Unknown
- Minimal Celery Version: N/A or Unknown
- Minimal Kombu Version: N/A or Unknown
- Minimal Broker Version: N/A or Unknown
- Minimal Result Backend Version: N/A or Unknown
- Minimal OS and/or Kernel Version: N/A or Unknown
- Minimal Broker Client Version: N/A or Unknown
- Minimal Result Backend Client Version: N/A or Unknown
Python Packages
pip freeze Output:
Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
Expected Behavior
Backend: file system
steriliser: pickle
Scenario: Define a group and retrieve the result, for instance, group([add.s(1,2), add.s(3,4)])().get()
This should give the correct result of two adding calculation
Actual Behavior
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv3/lib/python3.7/site-packages/celery/result.py:229: in get
on_message=on_message,
venv3/lib/python3.7/site-packages/celery/backends/base.py:579: in wait_for_pending
no_ack=no_ack,
venv3/lib/python3.7/site-packages/celery/backends/base.py:606: in wait_for
meta = self.get_task_meta(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:457: in get_task_meta
meta = self._get_task_meta_for(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:791: in _get_task_meta_for
return self.decode_result(meta)
venv3/lib/python3.7/site-packages/celery/backends/base.py:322: in decode_result
return self.meta_from_decoded(self.decode(payload))
venv3/lib/python3.7/site-packages/celery/backends/base.py:333: in decode
accept=self.accept)
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
return decode(data)
/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:130: in __exit__
self.gen.throw(type, value, traceback)
venv3/lib/python3.7/site-packages/kombu/serialization.py:54: in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
venv3/lib/python3.7/site-packages/vine/five.py:194: in reraise
raise value.with_traceback(tb)
venv3/lib/python3.7/site-packages/kombu/serialization.py:50: in _reraise_errors
yield
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
return decode(data)
venv3/lib/python3.7/site-packages/kombu/serialization.py:338: in unpickle
return pickle_loads(str_to_bytes(s))
venv3/lib/python3.7/site-packages/kombu/serialization.py:59: in pickle_loads
return load(BytesIO(s))
venv3/lib/python3.7/site-packages/celery/backends/base.py:69: in unpickle_backend
return cls(*args, app=current_app._get_current_object(), **kwargs)
venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:53: in __init__
path = self._find_path(url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <celery.backends.filesystem.FilesystemBackend object at 0x10b2c7c90>, url = None
def _find_path(self, url):
if not url:
> raise ImproperlyConfigured(E_NO_PATH_SET)
E kombu.exceptions.DecodeError: You need to configure a path for the file-system backend
venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:77: DecodeError
Solution
FilesystemBackend class defined in https://github.com/celery/celery/blob/master/celery/backends/filesystem.py#L34, misses __reduce__ method to pickle the instance properly. The following example would make it work.
def __reduce__(self, args=(), kwargs={}):
kwargs.update(
dict(url=self.url))
return super(FilesystemBackend, self).__reduce__(args, kwargs)