Skip to content

task/group chains fails in some scenarios #5467

@bartdw

Description

@bartdw

Checklist

  • I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the master branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the master branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Possible Duplicates

Environment & Settings

Celery version: 4.2.2 (windowlicker)
Celery report:
software -> celery:4.2.2 (windowlicker)
kombu:4.3.0 py:2.7.5
billiard:3.5.0.5 redis:3.2.1
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

broker_url: redis://localhost:6379/0
result_backend: redis://localhost:6379/0

Steps to Reproduce

Minimally Reproducible Test Case

from celery import signature, group, chain
from tasks import task

task1 = task.si('task1')
task2 = task.si('task2')

group1 = group(task.si('group1 job1'), task.si('group1 job2'))
group2 = group(task.si('group2 job1'), task.si('group2 job2'))

# working sequences
#res = chain(group1, group2)
#res = chain(group1, task1, group2)
#res = chain(task1, group1, task2, group2)
#res = chain(group1, group2, task1)

# failing sequence
res = chain(task1, group1, group2)

res.delay()

Expected Behavior

Tasks are passed to celery

Actual Behavior

It looks like celery is making the wrong decision to convert this chain into a chord. There are some workarounds available, but no real fix is available yet.

Traceback (most recent call last):
  File "debug_run.py", line 19, in <module>
    res.delay()
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 179, in delay
    return self.apply_async(partial_args, partial_kwargs)
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 557, in apply_async
    dict(self.options, **options) if options else self.options))
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 573, in run
    task_id, group_id, chord,
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 655, in prepare_steps
    task_id=prev_res.task_id, root_id=root_id, app=app,
AttributeError: 'GroupResult' object has no attribute 'task_id'

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions