Can I run a bot inside a Django project that will run independently of other Django apps?

Using Django Getting Started
1

I have a Django project for a website. And for the same website, I run another Python bot to scrap some data. I was wondering if I could somehow attach the bot to the Django project so that both can run on the same server without causing harm to each other. The bot and other Django apps should run independently of each other.

Does anyone know how this can be done?

Any help will be appreciated.

2

You can run external tasks on the same server, yes. Depending upon the precise nature of the script, you could run it as a cron job or as a persistent tasks using something like systemd or supervisord. If the scheduling is variable, you could also use Celery Beat to manage its execution.

It’s not really clear here what exactly you’re trying to do. On one hand, you wrote:

Later on however, you write:

So I’m not sure what you mean by “attaching the bot to the Django project.”

If you want it to have access to the same classes and models as your Django application, you would write your script as a custom management command.

Side note: It’s probably not necessary that it runs on the same server. There’s no requirement that those processes all run together on the same server. As long as they have access to the database, they can pretty much run anywhere.

1 Like
3

Thank you for your help. In the meantime I have made a fix. Here:

with the multiprocessing module I got a solution. In the manage.py file of a Django project, there’s an if statement at the end that just calls the main function defined in the same file.

I created two processes using the multiprocessing.Process class. One for the main function and one for the bot I wanted to attach to the project. Then I started those processes and the bot was running as well as other apps of the Django project.

here’s the code:

import multiprocessing

# A function to start the bot

def start_bot():
    ...

if __name__ == '__main__':
    bot_process = multiprocessing.Process(name='bot_process', target=start_bot)
    main_process = multiprocessing.Process(name='main_process', target=main)
    bot_process.start()
    main_process.start()

It’s working as expected but I am not sure if it’s the best solution.

4

You’ve just (partially) recreated what bash, systemd, runit, supervisord and Celery all do, without the features that those tools provide.

1 Like
5

Thank you. But is it a good practice? Will it cause any problem in the server or with the other things in Django?

6

No, it’s not good practice. It doesn’t provide any of the protections that one of the more robust solutions would provide. (For example, if one of the processes crashes for whatever reason, it’s not going to restart that process.)

It also doesn’t satisfy your stated requirement that:

in that they’re both subject to the status of the same parent process.

1 Like