-
-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Description
Checklist
-
I have verified that the issue exists against the
masterbranch of Celery. 90fe53f -
This has already been asked to the discussion group first.
-
I have read the relevant section in the
contribution guide
on reporting bugs. -
I have checked the issues list
for similar or identical bug reports. -
I have checked the pull requests list
for existing proposed fixes. -
I have checked the commit log
to find out if the bug was already fixed in the master branch. -
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
- I have included the output of
celery -A proj reportin the issue.
(if you are not able to do this, then at least specify the Celery
version affected). - I have verified that the issue exists against the
masterbranch of Celery. - I have included the contents of
pip freezein the issue. - I have included all the versions of all the external dependencies required
to reproduce this bug.
Optional Debugging Information
- I have tried reproducing the issue on more than one Python version
and/or implementation. - I have tried reproducing the issue on more than one message broker and/or
result backend. - I have tried reproducing the issue on more than one version of the message
broker and/or result backend. - I have tried reproducing the issue on more than one operating system.
- I have tried reproducing the issue on more than one workers pool.
- I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled. - I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
- None
Possible Duplicates
- None
Environment & Settings
Celery version:
celery report Output:
software -> celery:4.4.0 (cliffs) kombu:4.6.7 py:3.7.6
billiard:3.6.1.0 redis:3.3.11
platform -> system:Linux arch:64bit
kernel version:5.4.13-3-MANJARO imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://redis:6379/2
Steps to Reproduce
Required Dependencies
- Minimal Python Version: N/A or Unknown
- Minimal Celery Version: N/A or Unknown
- Minimal Kombu Version: N/A or Unknown
- Minimal Broker Version: N/A or Unknown
- Minimal Result Backend Version: N/A or Unknown
- Minimal OS and/or Kernel Version: N/A or Unknown
- Minimal Broker Client Version: N/A or Unknown
- Minimal Result Backend Client Version: N/A or Unknown
Python Packages
pip freeze Output:
amqp==2.5.2
apipkg==1.5
attrs==19.3.0
backcall==0.1.0
billiard==3.6.1.0
blinker==1.4
boltons==19.3.0
cachetools==4.0.0
celery==4.4.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Click==7.0
cryptography==2.8
decorator==4.4.1
defusedxml==0.6.0
dnspython==1.16.0
dotted==0.1.8
elasticsearch==7.1.0
elasticsearch-dsl==7.1.0
execnet==1.7.1
Flask==1.1.1
Flask-Caching==1.4.0
Flask-Cors==3.0.8
Flask-Limiter==1.1.0
Flask-Login==0.4.1
flask-mongoengine==0.9.5
flask-shell-ipython==0.4.1
Flask-WTF==0.14.2
gevent==1.4.0
google-api-core==1.15.0
google-auth==1.10.0
google-cloud-core==1.1.0
google-cloud-pubsub==1.1.0
google-cloud-storage==1.23.0
google-resumable-media==0.5.0
googleapis-common-protos==1.6.0
greenlet==0.4.15
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
httpagentparser==1.9.0
httplib2==0.15.0
idna==2.8
importlib-metadata==1.4.0
ipython==7.10.2
ipython-genutils==0.2.0
itsdangerous==1.1.0
jedi==0.15.2
Jinja2==2.10.3
kombu==4.6.7
libthumbor==1.3.2
limits==1.3
MarkupSafe==1.1.1
marshmallow==3.3.0
mixpanel==4.5.0
mmh3==2.5.1
mongoengine==0.18.2
more-itertools==8.1.0
multidict==4.7.4
ndg-httpsclient==0.5.1
newrelic==5.4.1.134
nexmo==2.4.0
oauth2client==4.1.3
oauthlib==3.1.0
packaging==20.0
parso==0.5.2
pexpect==4.7.0
phonenumbers==8.11.1
pickleshare==0.7.5
Pillow-SIMD==6.0.0.post0
pluggy==0.13.1
prompt-toolkit==3.0.2
protobuf==3.9.0
ptyprocess==0.6.0
pusher==2.1.4
py==1.8.1
py-cpuinfo==5.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybase62==0.4.3
pycparser==2.19
Pygments==2.5.2
PyJWT==1.7.1
pymongo==3.10.0
PyNaCl==1.3.0
pyOpenSSL==19.1.0
pyparsing==2.4.6
pytelegraf==0.3.3
pytest==5.3.2
pytest-benchmark==3.2.3
pytest-forked==1.1.3
pytest-mock==1.13.0
pytest-sugar==0.9.2
pytest-xdist==1.31.0
python-dateutil==2.8.1
python-rapidjson==0.9.1
python3-openid==3.1.0
pytz==2019.3
PyYAML==5.2
redis==3.3.11
requests==2.22.0
requests-oauthlib==1.3.0
rsa==4.0
semantic-version==2.8.3
sentry-sdk==0.13.5
six==1.13.0
social-auth-app-flask==1.0.0
social-auth-core==3.2.0
social-auth-storage-mongoengine==1.0.1
termcolor==1.1.0
traitlets==4.3.3
twilio==6.35.1
urllib3==1.25.7
uWSGI==2.0.18
vine==1.3.0
wcwidth==0.1.8
webargs==5.5.2
Werkzeug==0.15.5
WTForms==2.2.1
yarl==1.4.2
zipp==0.6.0
Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
import time
from flask import Flask
from celery import Celery, group
app = Flask(__name__)
celery = Celery('app', broker='redis://redis:6379/1', backend='redis://redis:6379/2')
@celery.task
def debug():
return
@app.route('/', methods={'GET', 'POST'})
def hello_world():
task = group([
debug.si() for i in range(10)
]).apply_async()
start = time.perf_counter()
task.get(timeout=5, interval=0.01)
print('END', (time.perf_counter() - start) * 1000)
return {}Expected Behavior
Scheduling noop tasks and setting interval should make the response time near the set interval in ideal clean environments.
Example: Setting task.get(interval=0.1) with 5x noop tasks, I would expect near 100ms response.
Actual Behavior
Regardless of the setting of interval, the response time is at least 500ms.
The cause is that interval is not passed all the way to get_many() where it defaults to 500ms which is where the minimum latency is comming from.
Lines 837 to 840 in cf82930
| def _iter_meta(self): | |
| return (meta for _, meta in self.backend.get_many( | |
| {r.id for r in self.results}, max_iterations=1, | |
| )) |
celery/celery/backends/base.py
Lines 663 to 666 in dc03b6d
| def get_many(self, task_ids, timeout=None, interval=0.5, no_ack=True, | |
| on_message=None, on_interval=None, max_iterations=None, | |
| READY_STATES=states.READY_STATES): | |
| interval = 0.5 if interval is None else interval |