Hello,
I want to stream financial data from Kafka over Django-Channels. How can I implement that?
Can I just write the consumer in the view or do I need a permanent background task, to connect to Kafka as a Consumer?
Thanks in advance
1 Like
What is the nature of this connection?
Are you connecting to it (“pull”) or is it opening a connection to you? (“push”)
What is the protocol being used across this connection? Is it a websocket, or some other protocol?
Once you get the data, what are you going to do with it? Are you storing it in a Model?
If it’s a pull, Channels may not be involved with that connection at all. (For example, I’ve got a project that opens a websocket to a server. As data is sent, my program writes the data to the database. Channels isn’t involved.)
You only need channels if you want a Django class to be the server-side of a persistent (e.g. websocket) connection.
(That’s discounting the possibility that you may want to use the Channels Layer as a lightweight task queue, but then there are other considerations involved.)
So I would use Kafka-Connect, which would be a TCP Connection. My Kafka Cluster would push the messages to me, which then would be directly send to the client over a websocket connection.
Is that enough information?
Please ask, if anything is not clarified yet
Getting there.
Is it one connection to Kafka to one client connection? Or is is one connection to Kafka being redistributed to ‘n’ connections?
Is your transmit policy “at least once” or “at most once”?
Do you need guaranteed delivery of any individual message at the client’s endpoint?
Do you need to retain the stored data locally?
Do you have timing constraints for delivery?
What sort of data volumes / number of clients are you dealing with?
What other architectural / configuration issues are relevant here?
1 Like
My goal is it to make it possible for my clients, that they can browse different stocks and other assets to see their pricing in real time. The price data should be stored in the db and then pushed as a message to Kafka. Django then gets these event message and push these messages to clients, who are subscribed in that specific asset topic. It would be nice if it’s scalable.
Thanks for your patience