Skip to content

worker with concurrency running only a single task while multiple tasks reserved  #6386

@kereg

Description

@kereg

Checklist

  • I have verified that the issue exists against the master branch of Celery.
  • This has already been asked to the discussion group first.
  • I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the master branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the master branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery Version: 5.0.0 (singularity)

celery report Output:

root@ad36e1cf0d81:/app# celery -A app.worker report

software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.7.5
            billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Linux arch:64bit
            kernel version:5.4.0-1025-aws imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

broker_url: 'amqp://prediction_celery:********@broker:5672//'
deprecated_settings: None
task_routes: {
 'app.worker.*': {'queue': 'high_memory_usage'}}
task_serializer: 'pickle'
accept_content: ['json', 'pickle']
broker_transport_options: {
    'interval_max': 0.5,
    'interval_start': 0,
    'interval_step': 0.2,
    'max_retries': 3}
worker_prefetch_multiplier: 1



software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.7.5
            billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Linux arch:64bit
            kernel version:5.4.0-1025-aws imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

deprecated_settings: None

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: 3.7.5
  • Minimal Celery Version: 5.0.0
  • Minimal Kombu Version: 5.0.2
  • Minimal Broker Version: RabbitMQ version: 3.8.9
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.1
bidalgo-common==0.0
billiard==3.6.3.0
boto3==1.10.45
botocore==1.13.45
celery==5.0.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
colormath==3.0.0
common==0.1
coverage==5.0.3
creative-analysis===-module.-creative-analysis.version-.from.-tmp-pip-req-build-v4szy4xr-creative-analysis-version.py-
custom-stats==0.1
cycler==0.10.0
dataclasses==0.6
dataclasses-json==0.3.7
decorator==4.4.2
docutils==0.15.2
idna==2.8
importlib-metadata==2.0.0
jmespath==0.10.0
joblib==0.16.0
kiwisolver==1.2.0
kombu==5.0.2
marshmallow==3.8.0
marshmallow-enum==1.5.1
matplotlib==3.1.1
mypy-extensions==0.4.3
mysqlclient==2.0.1
networkx==2.5
nlp-tools==0.0
nltk==3.4.5
numpy==1.16.5
pandas==0.25.1
patsy==0.5.1
Pillow==5.4.1
prompt-toolkit==3.0.7
pydantic==1.6.1
PyMySQL==0.9.3
pyparsing==2.4.7
python-dateutil==2.8.1
python-dotenv==0.10.3
pytz==2020.1
redis==3.3.11
requests==2.22.0
s3transfer==0.2.1
scikit-learn==0.21.3
scipy==1.3.1
six==1.15.0
SQLAlchemy==1.3.19
statsmodels==0.11.0
stringcase==1.2.0
text2digits==0.0.9
typing-extensions==3.7.4.3
typing-inspect==0.6.0
urllib3==1.25.10
vine==5.0.0
wcwidth==0.2.5
Werkzeug==1.0.1
zipp==3.2.0

Other Dependencies

Details

Status of node rabbit@f1b3c114b8d6 ... Runtime

OS PID: 275
OS: Linux
Uptime (seconds): 759
Is under maintenance?: false
RabbitMQ version: 3.8.9
Node name: rabbit@f1b3c114b8d6
Erlang configuration: Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:64]
Erlang processes: 669 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Minimally Reproducible Test Case

Details

task.py:

import os
from celery import Celery
from time import sleep

celery_app = Celery("worker", broker=os.environ['CELERY_BROKER_URL'])

@celery_app.task()
def run():
    sleep(60)
    return

run.py:

from task import run

if __name__ == '__main__':
    run.apply_async()
    run.apply_async()
  1. IMPORTANT - Start broker - bug happens only when first tasks run after broker starts.

  2. Run celery:
    $ celery -A task worker --pool=prefork --concurrency=2

  3. Run tasks:
    $ python run.py

  4. inspect active tasks:
    $ celery -A task inspect active

  5. inspect reserved tasks:
    $ celery -A task inspect reserved

Expected Behavior

Worker is running 2 active tasks

Actual Behavior

The worker is running only 1 active task and you can see 1 reserved task.

Same test performed on 4.4.7 (cliffs) worked as expected

Even in 5.0.0 when you run again

  1. Run tasks:
    $ python run.py

  2. inspect active tasks:
    $ celery -A task inspect active

  3. inspect reserved tasks:
    $ celery -A task inspect reserved

It will work as expected - you will see 2 active tasks and 0 reserved

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions