Hi, I am new to Django…and forums in general so hope creating a new topic was the right thing? I completed a few courses and am exploring with my first project. From the courses I understand how to develop a site where a user interacts with the database, adding products etc. However my project is regularly downloading sensor data from numerous APIs and storing these in a database. There is some processing of the raw data into separate tables. Then displaying this processed data in various ways to end users.
At the moment I have compiled an .exe that runs on a Windows scheduler (Windows server). Then I am using Django to present this data to the user with dashboards. I am planning to add a lot more apps over the coming months and again this part I am (reasonably) comfortable with…
The question is should I be thinking about bringing in the regular download and processing of the data into the Django project? If yes, then how would I do this?
Consider looking at creating your own management command:
From the article:
You may be asking yourself, how is that different from a regular Python script, or what’s the benefit of it. Well, the main advantage is that all Django machinery is loaded and ready to be used. That means you can import models, execute queries to the database using Django’s ORM and interact with all your project’s resources.
I then use a cron job/scheduled task to run that command at the interval I need.
I use management commands to regularly grab data from an external API and update my Django with it.
I also agree that using a management command and cron is the simplest approach in this case.
For completeness it would be good to mention that an alternative way to do this, is with a background worker that executes tasks from a queue. A Django integrated solution for this, is to use Celery with database backed scheduler like django-celery-beat. Using this setup, you will be able to define your tasks as part of your Django apps (if that is what you want), and define when to execute each task (scheduling) through Django admin. It does take effort to setup everything and have them working, but once you do it once, you will be able to reuse it in your other projects.
I should mention that Celery is rather extensive, with numerous configuration options that could be overwhelming. It maybe be considered as an overkill for simple tasks, but that depends the use case.
Thanks dstav. I am relatively new to programming, so using windows scheduler with the management command seems like the way forward at least in the short term. However I do like the idea of fully integrating into Django using celery and django-celery-beat. Will definitely come back to this when my head stops spinning from Django in general!