Hello everyone,
i want to ask a similar question to Using Kafka in Django but with some more details.
To begin with, my general goal is to create an IOT data streaming application that streams data i have from a CSV file into graps and save them into a database for later Machine learning processing (anomaly detection). Data are pushed into the app, i haven’t decided on transmit policy yet but since i have lots of data that are concurrent and similar i probably need at most once.
Right now my approach on implementing this app is more or less like a chat application, so that each machine has its own channel layer (chatroom) where each sensor subscribes using its parent machine id and sends its own data {id, value}. I want to use websocket protocol even though i am not sending data at such high rates that polling wont do the work.
I am confused in this following part. Since i want to save the data in a database for later processing do i have to feed a Model with the data? Should i use Channels for this or i can do it via Kafka/RabbitMQ/Redis (like the person asked on the question i tagged at the start).
Moreover, since i am using Celery for the background asynchronous tasks i am already using Redis and RabbitMQ as brokers but i cant understand how to feed data to my application and what is the way for the app to “see” the data that are streamed.
In conclusion my questions are these: Is the chat app an approach that makes sense? Can anybody explain how feed and fetch the data?
Any advice even a general one would be appreciated since i dont have much experience yet. Thanks in advance
Yes, Django / Channels would work for this - but it’s not the only way to do it.
Have you worked your way through the Channels tutorial? That shows you how to set up a consumer that listens to a websocket. Your consumer can receive the websocket data and then save it into a Model.
However, since this appears to be a one-way communication - from the external device to Django - I’m not sure I would use that method.
Your other options include:
-
Writing a network service that listens on a port, allowing an external connection and data transmission. That service could be written as a custom Django admin command, allowing the service to save data in models.
-
Allow the external device to write directly to redis, and write a custom Django admin command to monitor that redis key, taking action when data appears.
I’ve actually done all three in different projects.
Also, depending upon the rate and quantity of data, you could just submit it as data through an API-type call. There’s no real polling involved.
But only you can make the decision of which approach to take, and what is going to work best is going to depend upon a variety of factors. (Number of devices supplying this data, size and frequency of data submissions, the capability and software constraints of the sending device, networking infrastructure and constraints, what you’re looking to do with the data after you have it, etc, etc, etc)
First of all thanks for the answer, however i am still a bit confused.
First of all i havent worked with creating custon Django admin commands (apart from really basic stuff) so i am not sure how these solutions work. For example, i create a command that has as input the file from which i want to expose data to a network port? After exposing the data do i need a message broker?
Do you have any sources i can look into to understand better what you are saying to me?
I will search more the next days but any examples would be lifesaving
Thank you.
In Python, you can write a network server. (See Socket Programming in Python (Guide) – Real Python for example) The code for that would reside within the handle method of your custom command. Your IoT device could then connect to that server to send the necessary data.
You want to keep your mind open to all the different possible architectures. Don’t limit your thinking to http unless or until you’ve decided that it is the best option for your needs.
The point here is that you have a large number of different options. You shouldn’t let yourself get single-tracked into one line of thinking without understanding all the technical requirements and constraints of your requirements.