Celery Multi creates pid and log files in the wrong directory #6017
Description
Here is what docs say:
CELERYD_PID_FILE— Full path to the PID file. Default is/var/run/celery/%n.pidCELERYD_LOG_FILE— Full path to the worker log file. Default is/var/log/celery/%n%I.logNote: Using %I is important when using the prefork pool as having multiple processes share the same log file will lead to race conditions.
In reality, celery multi creates logs and pid files in the working directory instead of /var/run and /var/log:
--logfile=default%I.log --pidfile=default.pid
My systemd service:
# /etc/systemd/system/celery.service
[Unit]
Description=Celery Service
After=network.target
[Service]
Type=forking
User=myuser
Restart=always
EnvironmentFile=-/etc/conf.d/celery
WorkingDirectory=/home/myuser/myproject
ExecStart=/bin/bash -c 'source /etc/environment; \
/home/myuser/venv/bin/celery multi start ${CELERYD_NODES} \
-A ${CELERY_APP} ${CELERYD_OPTS}'
ExecStop=/bin/bash -c 'source /etc/environment; \
/home/myuser/venv/bin/celery multi stopwait ${CELERYD_NODES}'
ExecReload=/bin/bash -c 'source /etc/environment; \
/home/myuser/venv/bin/celery multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --loglevel="${CELERYD_LOG_LEVEL}" ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
My /etc/conf.d/celery:
# /etc/conf.d/celery
CELERYD_NODES="default slow"
CELERY_APP="pool.celery:app"
CELERYD_MULTI="multi"
CELERYD_OPTS="-Q:1 celery -Q:2 slow --time-limit=15 --concurrency=1 -O fair -E"
Systemd status report:
root@pool1:~# sudo systemctl status celery
● celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-03-31 13:38:02 UTC; 1min 23s ago
Process: 2648 ExecStop=/bin/bash -c source /etc/environment; /home/myuser/venv/bin/celery multi stopwait ${CELERYD_NODES} (code=exited, status=0/SUCCESS)
Process: 2661 ExecStart=/bin/bash -c source /etc/environment; /home/myuser/venv/bin/celery multi start ${CELERYD_NODES} -A ${CELERY_APP} ${CELERYD_OPTS} (code=exited, status=0/SUCCESS)
Main PID: 27107 (code=exited, status=1/FAILURE)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/celery.service
├─2680 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q celery --logfile=default%I.log --pidfile=default.pid --hostname=default@pool1.mycorp.
├─2688 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q slow --logfile=slow%I.log --pidfile=slow.pid --hostname=slow@pool1.mycorp.org
├─2694 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q celery --logfile=default%I.log --pidfile=default.pid --hostname=default@pool1.mycorp.
└─2695 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q slow --logfile=slow%I.log --pidfile=slow.pid --hostname=slow@pool1.mycorp.org
There are three options:
a. There's a bug in Celery 4.4.2
b. There's a mistake in docs
c. I'm missing something and everything works as expected
I'm using Celery 4.4.2 on Ubuntu 18.04.