Complex canvas might raise AttributeError: 'dict' object has no attribute 'clone' #5265
Description
WARNING: I'm still trying to collect all the necessary details and, if possible, provide a reproducible example. For the time, the error is happening constantly in one server, but doesn't happen in another one with the same source code.
I'm getting the following:
Traceback (most recent call last):
File "eggs/celery-4.2.1-py2.7.egg/celery/app/trace.py", line 439, in trace_task
parent_id=uuid, root_id=root_id,
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1232, in apply_async
return self.run(tasks, body, args, task_id=task_id, **options)
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1277, in run
header_result = header(*partial_args, task_id=group_id, **options)
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 953, in __call__
return self.apply_async(partial_args, **options)
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 978, in apply_async
args=args, kwargs=kwargs, **options))
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1054, in _apply_tasks
**options)
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 557, in apply_async
dict(self.options, **options) if options else self.options))
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 573, in run
task_id, group_id, chord,
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 683, in prepare_steps
task.link_error(errback)
File "eggs/celery-4.2.1-py2.7.egg/celery/canvas.py", line 1016, in link_error
sig = sig.clone().set(immutable=True)
AttributeError: 'dict' object has no attribute 'clone'
When trying to execute a complex canvas. The canvas itself is dynamically created. We have a single task that dispatches the actual calls. So all, jobs are instances of this task with different arguments. The task is: https://github.com/merchise-autrement/odoo/blob/merchise-10.0/odoo/jobs.py#L746
The canvas is constructed in: https://github.com/merchise/xopgi.base/blob/master/xopgi/xopgi_cdr/cdr_agent.py#L181
Basically, we have a layered graphs of events that depend on several evidences, which in turn depend on variables. We compute each variable in a job. Each variable job is linked to a group of jobs to update the dependent evidences. Each evidence job is liked to a group of event jobs. Each job has an on_error callback to signal the whole cycle had some errors.
I'm trying to detect if the issue comes from a particular workflow of jobs, or other cause.
I'm using celery 4.2.1, but I tested with master.