Django Channels with Redis PubSub

Using Django Async/Channels
21

The most direct and honest answer I can give is, “I don’t know what your “best” option would be”.

That’s a really interesting idea - an approach I wouldn’t have thought of. You would still need something to direct the traffic coming in to the right group, but that seems to be the easier of the two related parts to this.

You could also look at the RedisPubSubChannelLayer to see if/how this fits into your requirements.

My gut hunch is that the “optimal” solution may depend upon some combination of three factors:

  • The number of stocks (which you’ve identified as being in the neighborhood of 12,000)
  • The number of connections (users who are following these stocks)
  • The average number of stocks being watched by user

There are some effective upper limits in play here too - points at which you may need to scale horizontally, and potentially with a re-architecting of whatever you come up with as an initial design. (No, I have no idea what those limits may be - just that I’m willing to guarantee that they exist. Those limits are also going to depend a lot upon the platform on which it’s running and whether you have access to tune the kernel to handle a significant number of connections.)

As was recently said to me in a different post on a different topic, I think you’re probably in some uncharted waters here. You may be working in areas that no one has approached yet in a production deployment. (I’m personally not aware of any articles, papers, blog posts, etc detailing large-scale Channels deployments.) Whether you find that to be an issue is up to you.

The only way I know of to validate something of this nature would be to prototype it and test it at scale.

22

I will make a prototype for my idea. I can write you if you are interested.
So you have no company in mind, who is using Django-Channels at a larger production scale?!

23

I don’t personally know anyone - either company or individual - involved in what I would refer to as a “large scale” (1000+ concurrent connections) Channels production environment.

24

Ok, maybe I will separate my Websocket-Technology from my App. This would mean, that I continue to use Django as my backend, except the Websocket-Part, which then would be a different server (maybe Node.js).