Celery Beat is not a process manager. It’s an add-on for Celery. You would run it under systemd (or supervisord, etc) in a manner similar to Celery. It’s another, different, process like Celery.
Ok, I see that Celery Beat is a kind of simple dashboard for Cron tasks. That’s my impression after reading
https://django-celery-beat.readthedocs.io/en/latest/
I think I will not like it because it is not flexible enough like Cron. I believe I cannot schedule such tasks like “every Thursday before 26th of each month” or “run every second Friday after Easter”. Systemd is more flexible for that I believe.
Based on your experience it sounds that supervisord with docker is the right approach for me. The problem is that I have no knowledge of docker at all and implementing docker now when I am in the middle of implementing Celery would be to much mess 
Can be supervisord simply added to my current solution (Debian + Django + Apache with wsgi + Postgresql) without Docker at the moment?
You can’t (easily) run Celery tasks from Cron. The power of Celery Beat is that your schedule initiates Celery tasks needing to be run. For example, we have a Celery task to send emails every Friday morning. We use Beat to start that process.
Additionally, since the schedule is stored in the Database in Django (with the django-celery-beat package), you can programmatically manage the schedule within Django. For example, if you need to schedule something “every Thursday before the 26th of the month”, you could create a “Scheduling” Celery task that calculates the dates for the next month, and add those tasks to the schedule.
This provide a flexibility far exceeding what cron can deliver. (Oh, and Beat does support a cron-style schedule. My understanding is that it’s not quite as comprehensive as native cron, but it’s close enough for routine purposes. See django_celery_beat.models — django_celery_beat 2.5.0 documentation)
(And systemd is not a scheduler at all. It provides no facilities for running tasks on a scheduled basis. It starts tasks at boot time, and restarts them when necessary.)
Nope. It’s supervisord or docker. You don’t need them both and you don’t use them with each other. They are two separate and independent choices.
In my case, the decision between these two come down to how many different projects an individual server is going to support. If I’m going to run more than 2 or 3 projects on a server, using docker makes it a lot easier to manage those projects. Managing potentially-shared resources by more than that using supervisord can become an issue.
Making decision of the solution is one of the most important thing I believe.
About one year ago I decided to start using Django and it was the best decision ever. Now I would not live without Django.
That’s why so important for me is to pick up the right solution for running Celery in my environment. You have used almost all the available solutions and I believe feel what would be the best for me. As I mentioned I have currently Debian, Django, Apache, Wsgi, Postgresql, Celery, Scrapy. If You were me, what would you choose? Supervisor ?
Not really. It’s a pretty arbitrary decision, and not all that critical. Most importantly, the decision isn’t irrevocable - and it’s actually easy to switch between them.
This is what I refer to as an architectural decision - you have multiple choices with no distinct “right” or “wrong” answer. Your decision then should (in part) be based on non-technical factors.
You’re going to spend more time managing this system than configuring these elements.
Your requirements will be to check their status, and stop / restart when you do upgrades. You want the services to automatically restart when they fail or when the system is rebooted.
All these options will do that - reliably.
So from that perspective, start with what you’re comfortable using. If you’re comfortable with creating systemd service files, use it. If you’re more comfortable with a supervisord config file, go with that.
I think that’s made me to try the Celery Beat.
If I will choose Celery Beat for running Celery worker permanently, does it mean that Celeary Beat will run the task every period of time or it will make it real permanently (run it immediately) ?
Celery Beat is not needed for the application you have been working with here. It really shouldn’t be part of this discussion at all.
Your Django view places the task in the queue. Celery will dispatch that task to the right function.
Beat is used for something else entirely.
Please go back and reread post 57, and with that information in mind re-read the docs on running Celery as a persistent process.
Thank you. I will stick with systemd. I started reading what I need to do, but I am confused now how to do it unfortunately
I don’t understand the documentation Running the worker as a daemon — Celery 3.1.25 documentation
Would you help me please and tell what file I need to create first ?
To deamonize Celery I have created the following:
/etc/default/celeryd
# The names of the workers. This example create one worker
CELERYD_NODES="worker1"
# The name of the Celery App, should be the same as the python file
# where the Celery tasks are defined
CELERY_APP="app_celery"
# Log and PID directories
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Log level
CELERYD_LOG_LEVEL=INFO
# Path to celery binary, that is in your virtual environment
CELERY_BIN=/django/venv/bin/celery
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
/etc/systemd/system/celery.service
[Unit]
Description=Celery Service
After=network.target rabbitmq-server.service
Requires=rabbitmq-server.service
[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=/etc/default/celeryd
WorkingDirectory=/opt/celery
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always
[Install]
WantedBy=multi-user.target
/etc/tmpfiles.d/celery.conf
d /run/celery 0755 celery celery -
d /var/log/celery 0755 celery celery -
To activate the celery deamon service I run the following commands:
systemctl daemon-reload
To enable it at reboot:
systemctl enable celery.service
To start the service:
systemctl start celery.service
I have received the following problem:
systemctl status celery.service
× celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; enabled; preset: enabled)
Active: failed (Result: resources) since Thu 2023-12-14 00:41:07 CST; 2s ago
CPU: 0
Dec 14 00:41:07 n002 systemd[1]: celery.service: Scheduled restart job, restart counter is at 5.
Dec 14 00:41:07 n002 systemd[1]: Stopped celery.service - Celery Service.
Dec 14 00:41:07 n002 systemd[1]: celery.service: Start request repeated too quickly.
Dec 14 00:41:07 n002 systemd[1]: celery.service: Failed with result 'resources'.
Dec 14 00:41:07 n002 systemd[1]: Failed to start celery.service - Celery Service.
Please help.
This is a rather generic message telling you that your process is trying to use something that isn’t available.
Start by checking syslog - anything in there providing more information?
You’ve got directory definitions in tmpfiles.d, did you reboot? (Do those directories exist? Have the right permissions?)
Check the logs in /var/log/celery - any information in there?
Check the ownership and permissions to /opt/celery - ensure celery has access to it. (Also verify that /opt/celery is the project-directory for your Django project, and not the parent directory for it.)
Do you actually have a directory named django on root?
Directory /run/celery does not exists.
Directory /var/log/celery exists with permission 755 and there are some files there with errors but I am not sure if they are new or old Can I delete them all and wait for new logs?
Dictionary /opt/celery also does not exist. What is the folder /opt/celery for ?
I have /n/django/venv/bin/celery, just corrected it.
Yes, that’s one way to do it.
Or, you can read them to see what’s in there, looking for timestamps. (Most logs do timestamp their lines.)
That’s what you defined for:
That’s the current directory in which the commands are going to be issued.
I found the directory /opt/celery in example, so I thought it should be like that. My tasks are located in /n/django/app_name/tasks.py so I am confused what should be the directory for WorkingDirectory in my case.
I have deleted all logs from /var/log/celery, then rebooted system, run some tasks manually and there is nothing in /var/log/celery now
systemctl status celery.service
× celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Thu 2023-12-14 08:24:06 CST; 6min ago
Process: 1160 ExecStart=/bin/sh -c ${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FI>
CPU: 1ms
Dec 14 08:24:06 n002 systemd[1]: celery.service: Scheduled restart job, restart counter is at 5.
Dec 14 08:24:06 n002 systemd[1]: Stopped celery.service - Celery Service.
Dec 14 08:24:06 n002 systemd[1]: celery.service: Start request repeated too quickly.
Dec 14 08:24:06 n002 systemd[1]: celery.service: Failed with result 'exit-code'.
Dec 14 08:24:06 n002 systemd[1]: Failed to start celery.service - Celery Service.
If you were to run the celery worker from the command line, what directory would you be in? That is what your working directory should be for the service.
That would be /n/django I believe. The command I run is below:
/n/django# celery -A nweb worker -l INFO
I have changed it and now file celery.service looks like that:
[Unit]
Description=Celery Service
After=network.target rabbitmq-server.service
Requires=rabbitmq-server.service
[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=/etc/default/celeryd
WorkingDirectory=/n/django
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always
[Install]
WantedBy=multi-user.target
I need to mention that I did not create any user celery nor group celery .
Is that ok?
Check to see if that user/group exists. If they don’t, then no - it’s not ok. Both your tmpfiles.d directives and this service won’t function without them.