When calling self.replace() with a signature reconstructed from a serialized dictionary (e.g. after having been passed through a backend), if that signature is (or contains) a group which itself contains a chord, celery explodes after attempting to treat deeply nested dictionaries as signature objects. See below for a minimal repro I got together today.

My gut feel is that instantiating a signature from a dictionary may not be recursing down through the structure far enough and it leaves some of the encapsulated tasks as dicts. I've also noticed that groups containing a group also break in the same way, but I think that's because there's an internal promotion to a chord happening somewhere.

Checklist

• I have verified that the issue exists against the master branch of Celery.
• This has already been asked to the discussion group first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• Deserialize all tasks in a chain #4015

Possible Duplicates

• None

Environment & Settings

Celery version:

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

amqp==5.0.1
billiard==3.6.3.0
# Editable install with no version control (celery==5.0.0rc3)
-e /home/maybe/tmp/capp/venv/lib/python3.8/site-packages
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
future==0.18.2
kombu==5.0.2
prompt-toolkit==3.0.7
pytz==2020.1
redis==3.5.3
six==1.15.0
vine==5.0.0
wcwidth==0.2.5

Other Dependencies

Minimally Reproducible Test Case

import celery

app = celery.Celery("app", backend="redis://")

@app.task
def foo(*_):
    return 42

@app.task(bind=True)
def replace_with(self, sig):
    assert isinstance(sig, dict)
    sig = celery.Signature.from_dict(sig)
    raise self.replace(sig)

if __name__ == "__main__":
    sig = celery.group(
        celery.group(foo.s()),
    )
    res = sig.delay()
    print(res.get())

    sig.freeze()
    res = replace_with.delay(sig)
    print(res.get())

Expected Behavior

It shouldn't explode. Presumably tasks within the group/chord should be signatures rather than dicts.

Actual Behavior

Stack trace in the worker output:

[2020-09-08 12:44:05,453: DEBUG/MainProcess] Task accepted: app.replace_with[dcea02fd-23a3-404a-9fdd-b213eb51c0d1] pid:453431
[2020-09-08 12:44:05,457: ERROR/ForkPoolWorker-8] Task app.replace_with[dcea02fd-23a3-404a-9fdd-b213eb51c0d1] raised unexpected: AttributeError("'dict' object has no attribute '_app'")
Traceback (most recent call last):
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/kombu/utils/objects.py", line 41, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'app'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/app/trace.py", line 409, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/app/trace.py", line 701, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/maybe/tmp/capp/app.py", line 13, in replace_with
    raise self.replace(sig)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/app/task.py", line 894, in replace
    sig.freeze(self.request.id)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/canvas.py", line 1302, in freeze
    self.tasks = group(self.tasks, app=self.app)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/canvas.py", line 1456, in app
    return self._get_app(self.body)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/canvas.py", line 1466, in _get_app
    app = tasks[0]._app
AttributeError: 'dict' object has no attribute '_app'