Celery works on dev/runserver, not on Apache (WSGI)

Using Django Deployment
1

Hi!

Happy Django user here. Implementing Celery as we speak. It works well if I use runserver, but apache does not work well… I have restarted my workers often, but keep seeing “celery.exceptions.NotRegistered: ‘lightning.tasks.test_task’” in my apache error logs.

For some reason, the apache “environment” does not see/detect my celery tasks, while ./manage.py runserver does work.

Some background:

  • Celery using default RabbitMQ backend, running with debug shows me the worker gets the task:
[tasks]
  . test_app.tasks.test_task
  • Apache uses WSGI, the wsgi file:
import os
import sys               
sys.path.append('/opt/web_proj/webfrontend')              
sys.path.append('/opt/web_proj')        
os.environ["CELERY_LOADER"] = "django"
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'web_proj.settings')
application = get_wsgi_application()
  • Error log:
[Mon Sep 26 13:00:35.316682 2022] [wsgi:error] [pid 45554:tid 140694624315136] [remote 10.0.60.16:61547] celery.exceptions.NotRegistered: 'test_app.tasks.test_task'

All help much appreciated. Have messed with this for 2 hours now…

2

If you still need help with this, what settings file is used by your celery worker processes and by manage.py?

3

Hi! I don’t have a settings file for celery… I run: python3 -m celery --app=web_proj worker --beat -l info. I set cwd to the root of my project.

By now I have switched Apache to Hypercorn, which works well with celery. So not investigating much further. Thanks for your reply.

4

This can happen when there are instances of celery workers running old/different code that you may not be aware of.
Run ps A | grep celery and you may find there are processes trying to process something from the queue, but they don’t have the method you are calling, and hence you get the NotRegistered error.
Another strange indicator is when you add a new parameter to a method and you may sporadically get a message like:
TypeError: execute_task() takes 6 positional arguments but 7 were given
If sometimes it works, and sometimes it doesn’t, you likely have an old instance of your code grabbing items off your queue and failing.