Signal to execute call_command with Apache wsgi

Using Django Deployment
1

Hello all,

I am newbie here :slight_smile: so please let me know if my question not follow all the rules here. I appreciate it.

I have a problem with calling Django command using signal. Here is my code and settings.

signals.py

from django.dispatch import Signal
app_library_number = Signal()

receivers.py

@receiver(post_save, sender=Library)
def app_library_number_handler(sender, instance, *args, **kwargs):
    call_command('app_library_number', '--number', instance.number)

The command itself is to run scrapy crawler I present below.

management / commands / app_library_number.py

class Command(BaseCommand):
    help = 'Run app_library_number.py'
    def add_arguments(self, parser):
        parser.add_argument('--number', help='Library Number')

    def handle(self, *args, **options):
        number = options['number']
        process = CrawlerProcess(get_project_settings())
        process.crawl(LibraryNumber, number)
        process.start()

All works fine if I call the command via shell command as follows:

`python3.11 manage.py app_library_number --number 12345678`

but when I run it via signal in Django I am getting error :frowning:

Exception Type: ValueError
Exception Value: set_wakeup_fd only works in main thread of the main interpreter

For versions I use Django ver. 4.1.3 with Apache 2.4.57 (with wsgi)

I appreciate any help,
Lexe

2

Side Note: When posting code or templates here, surround the code (or template) between lines of three backtick - ` characters. This means you’ll have a line of ```, then your code, then another line of ```. This forces the forum software to keep the text properly formatted.
(I’ve taken the liberty of fixing your original post for this.)

3

You don’t want to do this, not this way, and especially not in a signal.

Use the right tool here - set this up as a Celery task and spawn it off from Celery.

4

Thank you KenWhitesell for letting me know. I will try to format the questions correctly to build the knowledge database clear and available for others.

Thank you

5

I understand a Celery is a scheduler like cron, but I do not want run the command periodically, just after my model Library is saved I would like to run external command via signal.

Is it Celery still for that?

6

That is a misunderstanding. While you can use it for scheduled tasks, that’s actually not its original design intent.

It’s a task queue, which is what you’re looking for here.

Also, you don’t need to use a signal here - and it’s probably to your advantage to not use one.

7

Mhhh, the problem I have with Celery is that I use Signals for performing other tasks but cannot with Scrapy. I feel the Celery is like using gun to kill a fly. Am I wrong?

8

Yes you are wrong.

I’m getting the impression that you’re thinking that signals are an asynchronous operation, when they are not. A Django signal is a function call, like any other, causing the signal code to be called in line when the signal is fired.

There are very few situations where signals are advantageous, and never provide a net benefit when you have control of the situation when the signal would be triggered.

You especially never want to spawn off an external process from within your main line of Django code. That’s a problem waiting to occur. (This is a significant anti-pattern that has been addressed here multiple times in the past.)

9

Wow, Thank you so much for your answer.

You are right, I was thinking that signals can handle the external command asynchronously.
I believe I have to read now about Celery and clean up my code.

I know it is not the right place, but can you recommend any tutorial how to get familiar with Celery in the expedited way?

10

The official “Getting Started” page in the docs is a good place to start.

See First Steps with Celery — Celery 5.3.6 documentation

1 Like
11

Thank you so much Ken. I appreciate your prompt help and time.
Let me look at the Celery. I hope I will be able to include it to my project.

12

The installation was very easy :slight_smile:
I am now new user of Celery :slight_smile:
I am going further with my Celery journey. Thank you a lot.

13

I am going thru Celery documentation and understand the Celery can be installed “inside” my Django app or as external app, so I can use it for many apps in case of large project.

I believe it is better to install as a separate app, am I correct?

14

Where the Celery task is defined does not limit the apps that can use it.

If your task primarily uses models (or other resources) from one specific app, then that app is probably the best place to put the task code. Otherwise, the task code can be located anywhere.

1 Like
15

Ken, Celery is amazing :slight_smile:

Here is what I did following your suggestions

  1. I removed my signals as they are wrong approach here
  2. Installed Task queue (Celery)
    pip3.11 install celery
  3. Installed RabbitMQ (Message Broker)
    apt-get install rabbitmq-server
  4. In file celery.py initiated Celery configuration

app = Celery('app_celery',
             broker='amqp://',
             backend='rpc://',
             include=['app_celery.tasks'])

app.conf.update(
    result_expires=3600,
)

if __name__ == '__main__':
    app.start()
  1. In file tasks.py prepared task that call Scrapy parser via Django command
from app_celery.celery import app
from django.core.management import call_command


@app.task
def app_scrapy_library_number(number):
    call_command('app_scrapy_library_number', '--number', number)
  1. Initiated Celery task in save() method of Library model
    def save(self, *args, **kwargs):
        queue = "app_scrapy_library_number number={} name='{}'".format(self.number, self.name)
        app_scrapy_library_number.apply_async((self.number,), queue=queue)

        return super().save(*args, **kwargs)
  1. Now when I save() my Library model I see the task
  2. Command
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged
  1. Result
    app_scrapy_library_number number=857324564 name='Capitan Ocean' 1 1 0

So, all seems to looks ok, except I do not know how to automatically execute the tasks in queue ‘app_scrapy_library_number’ :frowning:

16

I think I stuck bringing Celery to work :frowning:

It seems that RabbitMq works because when Django initiate task:
app_scrapy_library_number.apply_async((instance.number,), queue='my_queue')
it goes to RabbitMq and I see it:

Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name    messages        messages_ready  messages_unacknowledged
my_queue 1       1       0

I believe the task from the rabbitmq queue should automatically go to Cellery, but it does not go :frowning: Any suggestion what I set up wrongly

17

Are you running the Celery Worker task? If so, how? (What is the command line you’re using?)

See Running the Celery worker server and Workers Guide — Celery 5.3.6 documentation

18

I was not running any Celery worker task so far, but your question suggested me to run
celery -A app_celery worker -l INFO

and I run it now. Here is the result:

[2023-12-12 18:32:46,523: ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@hostname:15672//: timed out.
Trying again in 8.00 seconds... (4/100)

I understand my RabbitMq url is wrong.

19

Mhh, should the port be 15672 or 25672 or 5672 ?

When I run command:

ss -antpl | grep 15672
LISTEN 0      1024         0.0.0.0:15672      0.0.0.0:*    users:(("beam.smp",pid=1559,fd=40))

and when run:

ss -antpl | grep 5672
LISTEN 0      1024         0.0.0.0:15672      0.0.0.0:*    users:(("beam.smp",pid=1559,fd=40))
LISTEN 0      128          0.0.0.0:25672      0.0.0.0:*    users:(("beam.smp",pid=1559,fd=18))
LISTEN 0      128                *:5672             *:*    users:(("beam.smp",pid=1559,fd=41))
20

I corrected port to 5672 and now the command gives the following:

celery multi start worker1 -A app_celery -l INFO
celery multi v5.3.6 (emerald-rush)
> Starting nodes...
        > worker1@hostname: OK

The tasks in MQ are now like that:

rabbitmqctl list_queues name messages messages_ready messages_unacknowledged
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name    messages        messages_ready  messages_unacknowledged
celery  0       0       0
worker1@hostname.celery.pidbox      0       0       0
my_queue 8       8       0

so Django is addding tasks to “my_queue” and Celery is listening on “worker1@hostname.celery.pidbox” ??? How to make them to talk?