Skip to content

Add a configuration option to disable prefetch completely #6500

New issue
New issue
Closed
#7305
Closed
Add a configuration option to disable prefetch completely#6500
#7305
Labels
Component: Celery MasterComponent: ConfigurationIssue Type: Feature Request
Milestone
5.6.0
@george-miller

Description

@george-miller
george-miller
opened on Nov 24, 2020

Checklist

  • I have verified that the issue exists against the master branch of Celery.
  • This has already been asked to the discussion group first.
  • I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the master branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the master branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version: 4.3.0 (rhubarb)

celery report Output: 

software -> celery:4.3.0 (rhubarb) kombu:4.6.11 py:3.6.12
            billiard:3.6.3.0 redis:3.3.11
platform -> system:Linux arch:64bit, ELF
            kernel version:4.14.106-97.85.amzn2.x86_64 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://query-runner-redis-master:6379/0

BROKER_URL: 'redis://query-runner-redis-master:6379/0'
CELERY_RESULT_BACKEND: 'redis://query-runner-redis-master:6379/0'
CELERY_INCLUDE: ['tasks.query']
CELERY_TIMEZONE: 'UTC'
CELERY_TASK_SERIALIZER: 'json'
CELERY_IGNORE_RESULT: True
CELERY_TASK_ACKS_LATE: True
CELERYD_PREFETCH_MULTIPLIER: 1
CELERY_ROUTES: {
    'tasks.query.kill_query': {'queue': 'meta'},
    'tasks.query.run_query': {'queue': 'default'}}
CELERY_ALWAYS_EAGER: False
CELERYBEAT_SCHEDULE: {
 }
CELERY_TASK_CREATE_MISSING_QUEUES: True

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: N/A or Unknown
  • Minimal Celery Version: N/A or Unknown
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output: 

amqp==2.6.1
billiard==3.6.3.0
blinker==1.4
boto==2.49.0
boto3==1.16.21
botocore==1.19.21
celery==4.3.0
celery-prometheus-exporter==1.7.0
certifi==2020.11.8
chardet==3.0.4
cheroot==8.4.7
CherryPy==18.6.0
click==7.1.2
datadog==0.39.0
decorator==4.4.2
env-excavator==1.5.0
Flask==1.1.2
Flask-Cors==3.0.9
flask-swagger==0.2.14
flask-swagger-ui==3.36.0
flower==0.9.5
future==0.18.2
gunicorn==20.0.4
humanize==3.1.0
idna==2.10
importlib-metadata==2.0.0
importlib-resources==3.3.0
inflection==0.5.1
influxdb==5.3.1
itsdangerous==1.1.0
jaraco.classes==3.1.0
jaraco.collections==3.0.0
jaraco.functools==3.0.1
jaraco.text==3.2.0
JayDeBeApi==1.1.1
Jinja2==2.11.2
jmespath==0.10.0
JPype1==0.6.3
kombu==4.6.11
MarkupSafe==1.1.1
more-itertools==8.6.0
msgpack==1.0.0
numpy==1.17.0
portend==2.7.0
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
psycopg2==2.8.6
pycrypto==2.6.1
PyHive==0.6.3
PyMySQL==0.10.1
python-dateutil==2.8.1
pytz==2020.4
PyYAML==5.3.1
qds-sdk==1.16.1
raven==6.10.0
redis==3.3.11
redlock==1.2.0
requests==2.25.0
s3transfer==0.3.3
six==1.15.0
tempora==4.0.1
thrift==0.13.0
tornado==6.1
urllib3==1.26.2
vine==1.3.0
Werkzeug==1.0.1
zc.lockfile==2.0
zipp==3.4.0

Other Dependencies

Details

N/A

Minimally Reproducible Test Case

Details 







Expected Behavior



I was hoping by specifying -Ofair that tasks would not be reserved by workers as stated here https://docs.celeryproject.org/en/v4.3.0/userguide/optimizing.html?highlight=optimization#prefork-pool-prefetch-settings


Actual Behavior



Workers have reserved tasks while other workers are free to do work.


I am going to paste the state we are seeing in prod when a lot of tasks come in.  Notice how query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj has a reserved task while celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9 is empty and could run that task.  Please note the name of the worker is the name of the queue they are reading from.



celery inspect active and celery inspect reserved Output:




root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect active
2020-11-24 21:03:59,312 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-> celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    * {'id': '93a32a04-7a6d-4a32-a2ee-8b88ca4f9049', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_aa42f1c1970bc1ac2778a65c60b2e08b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3107', 'caller_json': {'queryId': 3107, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251603.8548772, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '747a1dd7-209a-4870-a636-c18176587723', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_373fafa1388d047c83cae9d40f249655', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3149 on dash 616', 'caller_json': {'queryId': 3149, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251609.6809852, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': '28ec904f-197f-4aef-99e2-0887f6f85ec7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_829e5787764013fbf5eb0958eec86a8f', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2400 on dash 609', 'caller_json': {'queryId': 2400, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251604.9346218, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '705d4f46-662d-4930-9011-265e1065346d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_04cd73d13695b2a7d9503148018c2650', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3179 on dash 616', 'caller_json': {'queryId': 3179, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606240820.285725, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-> celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    * {'id': '7afd7a99-368e-4995-8e5a-a1802dff7476', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9fe008b693e6dc12999bf58212430afa', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2403 on dash 609', 'caller_json': {'queryId': 2403, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251605.1041203, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '4d59fc3f-40b9-4d51-ad41-e241518c629f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9a6f5ab9f25ab21bddcc128aee8677d6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2396 on dash 609', 'caller_json': {'queryId': 2396, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251604.2693179, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '05bc6789-2183-47d2-abe0-7093bad200a5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_a937a6def4c1010c15dd3c5e1b488d2d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3116 on dash 616', 'caller_json': {'queryId': 3116, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251609.2829497, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    * {'id': 'd6ddf309-e58a-4f2d-be71-1824e618716e', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b7253439941188da3776ffdc8bab2b07', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2401 on dash 609', 'caller_json': {'queryId': 2401, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251605.0180476, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '62bb08c5-d6a6-4966-8de1-0d872faba09f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b75d10ba197e1bb1ab787217ca5daa96', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2395 on dash 609', 'caller_json': {'queryId': 2395, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251604.1448753, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '82240c16-ec84-4279-9cd4-694a73a145eb', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6435b6004863fd5a10de1abc8bf294cd', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3188 on dash 624', 'caller_json': {'queryId': 3188, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.9815636, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '9c062ae6-bc8d-44d1-b861-068cd1032c38', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6912fef8aee81d8aa0a68f1326e5bbf6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3117 on dash 616', 'caller_json': {'queryId': 3117, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.2058136, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    * {'id': '87228aca-e02f-48a3-9096-5540c7b4fb22', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e97a5e28903b3652a4a2c386fc287cc6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3115 on dash 616', 'caller_json': {'queryId': 3115, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4', 'time_start': 1606251609.4260392, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    * {'id': '1a8982d5-32cd-4fcf-a134-72a79d7e5da7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_02758ef185f77a7397ead9e8e90e9743', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2397 on dash 609', 'caller_json': {'queryId': 2397, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251604.4064894, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'b2804139-6d51-4ffe-817b-8b9efc7a9cd4', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_1920e1f67f84ac24fb3d2b8e8bb4bd22', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3114 on dash 616', 'caller_json': {'queryId': 3114, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251609.3467886, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-> celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    * {'id': '741f2751-ca24-4bcd-b78c-1d92e679952d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e526e7f0e7a7c28271dc8f08ee19504b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3021 on dash 595', 'caller_json': {'queryId': 3021, 'interval': 'HOURLY', 'dashboard_id': 595, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj', 'time_start': 1606251616.69369, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
-> celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    * {'id': '650009cc-4d6b-42c4-96b1-9a3112a1e5c8', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_711f6cdfe13c22710887f00f5678baeb', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2399 on dash 609', 'caller_json': {'queryId': 2399, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251604.6513581, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': 'd74e3272-45f0-44ee-bcab-7b67de60b407', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_5ca350ca8b4b185ad42f94c6dd3b6fc9', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3189 on dash 624', 'caller_json': {'queryId': 3189, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251611.4904687, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '6f61b237-fa9e-4b30-8131-8d8ee75306c5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9bc92aabee38e8d6d607bbb24785d814', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2137 on dash 557', 'caller_json': {'queryId': 2137, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606248030.1126661, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': 'ab45784f-8d85-452c-b690-43ead1a010c6', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_bb8a680bab37f6bc86eb9f5c7435d656', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2816 on dash 609', 'caller_json': {'queryId': 2816, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251605.576281, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '4e64157e-9356-4362-88af-318013c0c4de', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4859064fc4b96df99eec68e94f733157', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3143 on dash 616', 'caller_json': {'queryId': 3143, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251609.5426545, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': 'ac0c403b-ade6-4377-9a41-6d305268204b', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b04a82ce9376c9c6618c3a300abe4182', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3109', 'caller_json': {'queryId': 3109, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606240815.8285584, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'e9c368de-d47f-4a84-b05d-81cbda00a394', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4d03e965dbd57a05d6071c0b5cbf699d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2398 on dash 609', 'caller_json': {'queryId': 2398, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251604.517987, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-> celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    * {'id': '03e786df-44b1-412a-8354-3acbc762528f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_0687aa16794011f4d1a0ee1023173bf8', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3105 on dash 557', 'caller_json': {'queryId': 3105, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251618.9102547, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '646617ce-9835-4b60-b5fd-c4a4a5631749', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_ed09cf8c1166115a4ed1a5aa33b032c7', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3144 on dash 616', 'caller_json': {'queryId': 3144, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251609.6082683, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect reserved
2020-11-24 21:04:03,851 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:04:03,854 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:04:03,855 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-> celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-> celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-> celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': '3f501bbe-6c56-48e2-ba64-9b1adac30695', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6eb04958c42a71ebd014f9be9e116f73', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2823 on dash 557', 'caller_json': {'queryId': 2823, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': None}
-> celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    - empty -
-> celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -








Also, here is the output of celery inspect stats where you can see fair is getting in properly



celery inspect stats for worker query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9  Output:


```
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    {
        "broker": {
            "alternates": [],
            "connect_timeout": 4,
            "failover_strategy": "round-robin",
            "heartbeat": 120.0,
            "hostname": "query-runner-redis-master",
            "insist": false,
            "login_method": null,
            "port": 6379,
            "ssl": false,
            "transport": "redis",
            "transport_options": {},
            "uri_prefix": null,
            "userid": null,
            "virtual_host": "0"
        },
        "clock": "8600271",
        "pid": 8,
        "pool": {
            "max-concurrency": 4,
            "max-tasks-per-child": "N/A",
            "processes": [
                13,
                14,
                15,
                16
            ],
            "put-guarded-by-semaphore": false,
            "timeouts": [
                0,
                0
            ],
            "writes": {
                "all": "42.86%, 42.86%, 14.29%",
                "avg": "33.33%",
                "inqueues": {
                    "active": 0,
                    "total": 4
                },
                "raw": "3, 3, 1",
                "strategy": "fair",
                "total": 7
            }
        },
        "prefetch_count": 4,
        "rusage": {
            "idrss": 0,
            "inblock": 72400,
            "isrss": 0,
            "ixrss": 0,
            "majflt": 158,
            "maxrss": 68800,
            "minflt": 69557,
            "msgrcv": 0,
            "msgsnd": 0,
            "nivcsw": 2942,
            "nsignals": 0,
            "nswap": 0,
            "nvcsw": 334278,
            "oublock": 1584,
            "stime": 7.775837,
            "utime": 152.889586
        },
        "total": {
            "tasks.query.run_query": 7
        }
    }
```





Generally, we have a lot of tasks long running tasks.  This issue is making our tasks take double the amount of time because they are unnecessarily waiting behind other tasks without utilizing the other workers.
Metadata
Metadata
Assignees
No one assigned
    Labels
    Component: Celery MasterComponent: ConfigurationIssue Type: Feature Request
    Type
    No type
    Projects
    No projects
    Milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions