-
-
Notifications
You must be signed in to change notification settings - Fork 5k
Description
Checklist
- I have verified that the issue exists against the
masterbranch of Celery. - This has already been asked to the discussion group first.
- I have read the relevant section in the
contribution guide
on reporting bugs. - I have checked the issues list
for similar or identical bug reports. - I have checked the pull requests list
for existing proposed fixes. - I have checked the commit log
to find out if the bug was already fixed in the master branch. - I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
- I have included the output of
celery -A proj reportin the issue.
(if you are not able to do this, then at least specify the Celery
version affected). - I have verified that the issue exists against the
masterbranch of Celery. - I have included the contents of
pip freezein the issue. - I have included all the versions of all the external dependencies required
to reproduce this bug.
Optional Debugging Information
- I have tried reproducing the issue on more than one Python version
and/or implementation. - I have tried reproducing the issue on more than one message broker and/or
result backend. - I have tried reproducing the issue on more than one version of the message
broker and/or result backend. - I have tried reproducing the issue on more than one operating system.
- I have tried reproducing the issue on more than one workers pool.
- I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled. - I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
- None
Possible Duplicates
- None
Environment & Settings
Celery Version: 5.0.0 (singularity)
celery report Output:
root@ad36e1cf0d81:/app# celery -A app.worker report
software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.7.5
billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Linux arch:64bit
kernel version:5.4.0-1025-aws imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
broker_url: 'amqp://prediction_celery:********@broker:5672//'
deprecated_settings: None
task_routes: {
'app.worker.*': {'queue': 'high_memory_usage'}}
task_serializer: 'pickle'
accept_content: ['json', 'pickle']
broker_transport_options: {
'interval_max': 0.5,
'interval_start': 0,
'interval_step': 0.2,
'max_retries': 3}
worker_prefetch_multiplier: 1
software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.7.5
billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Linux arch:64bit
kernel version:5.4.0-1025-aws imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled
deprecated_settings: None
Steps to Reproduce
Required Dependencies
- Minimal Python Version: 3.7.5
- Minimal Celery Version: 5.0.0
- Minimal Kombu Version: 5.0.2
- Minimal Broker Version: RabbitMQ version: 3.8.9
- Minimal Result Backend Version: N/A or Unknown
- Minimal OS and/or Kernel Version: N/A or Unknown
- Minimal Broker Client Version: N/A or Unknown
- Minimal Result Backend Client Version: N/A or Unknown
Python Packages
pip freeze Output:
amqp==5.0.1
bidalgo-common==0.0
billiard==3.6.3.0
boto3==1.10.45
botocore==1.13.45
celery==5.0.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
colormath==3.0.0
common==0.1
coverage==5.0.3
creative-analysis===-module.-creative-analysis.version-.from.-tmp-pip-req-build-v4szy4xr-creative-analysis-version.py-
custom-stats==0.1
cycler==0.10.0
dataclasses==0.6
dataclasses-json==0.3.7
decorator==4.4.2
docutils==0.15.2
idna==2.8
importlib-metadata==2.0.0
jmespath==0.10.0
joblib==0.16.0
kiwisolver==1.2.0
kombu==5.0.2
marshmallow==3.8.0
marshmallow-enum==1.5.1
matplotlib==3.1.1
mypy-extensions==0.4.3
mysqlclient==2.0.1
networkx==2.5
nlp-tools==0.0
nltk==3.4.5
numpy==1.16.5
pandas==0.25.1
patsy==0.5.1
Pillow==5.4.1
prompt-toolkit==3.0.7
pydantic==1.6.1
PyMySQL==0.9.3
pyparsing==2.4.7
python-dateutil==2.8.1
python-dotenv==0.10.3
pytz==2020.1
redis==3.3.11
requests==2.22.0
s3transfer==0.2.1
scikit-learn==0.21.3
scipy==1.3.1
six==1.15.0
SQLAlchemy==1.3.19
statsmodels==0.11.0
stringcase==1.2.0
text2digits==0.0.9
typing-extensions==3.7.4.3
typing-inspect==0.6.0
urllib3==1.25.10
vine==5.0.0
wcwidth==0.2.5
Werkzeug==1.0.1
zipp==3.2.0
Other Dependencies
Details
Status of node rabbit@f1b3c114b8d6 ... Runtime
OS PID: 275
OS: Linux
Uptime (seconds): 759
Is under maintenance?: false
RabbitMQ version: 3.8.9
Node name: rabbit@f1b3c114b8d6
Erlang configuration: Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:64]
Erlang processes: 669 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60
Minimally Reproducible Test Case
Details
task.py:
import os
from celery import Celery
from time import sleep
celery_app = Celery("worker", broker=os.environ['CELERY_BROKER_URL'])
@celery_app.task()
def run():
sleep(60)
return
run.py:
from task import run
if __name__ == '__main__':
run.apply_async()
run.apply_async()
-
IMPORTANT - Start broker - bug happens only when first tasks run after broker starts.
-
Run celery:
$ celery -A task worker --pool=prefork --concurrency=2 -
Run tasks:
$ python run.py -
inspect active tasks:
$ celery -A task inspect active -
inspect reserved tasks:
$ celery -A task inspect reserved
Expected Behavior
Worker is running 2 active tasks
Actual Behavior
The worker is running only 1 active task and you can see 1 reserved task.
Same test performed on 4.4.7 (cliffs) worked as expected
Even in 5.0.0 when you run again
-
Run tasks:
$ python run.py -
inspect active tasks:
$ celery -A task inspect active -
inspect reserved tasks:
$ celery -A task inspect reserved
It will work as expected - you will see 2 active tasks and 0 reserved