How to maintain long connection to redis?

Using Django Getting Started
1

Hi everyone,
I’m using Redis as a message broker, and thus I want to keep a connection with it throughout the runtime of my Django application, so that I can respond to an incoming message from a service as soon as possible. I’d like to use the BLPOP redis command (blocking pop), which waits until a message is available in a given queue as soon as possible.

But I’m wondering how to incorporate this into my Django app, because directly in the Django app it seems weird (I don’t even know where to place the starting and killing of this long-lasting connection), it doesn’t seem like the best place to put it in Celery either.

Just for info, I use Django v4.2 (run by Gunicorn), Postgres DB, Celery (to manage tasks) and Redis.

Any ideas?

As a proof of concept I used a celery task that ran for 9s and was started every 10s, but it’s not an elegant solution, so I would like to NOT use it in production, if you understand.

I would be delighted with any response.
Thank you for your time!

2

Hello there.
If what you want is only, and only one persistent connection with redis, and this will be hanging, waiting for the redis response, then I would write a python script, that would do that. Something like this:

import django
import os

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "some_project.settings")
django.setup()

def main():
  # pseudo-code on this block
  redis.connect()
  while True:
    value = redis.blpop("some_key")
    # do something with it
    # teardown the current execution, like waiting for a bit, or just let it wait again
    

if __name__ == "__main__":
  main()

And of course, you would start this python script alongside your stack of processes (django,db, etc).

2 Likes
3

Welcome @torx-cz !

To add to the excellent answer above, I’d also suggest that you use something like systemd, supervisord, or runit to manage the execution of that script and to ensure that it stays running.

2 Likes
4

Thank you a lot for your ideas and responses.

@leandrodesouzadev Thank you for this idea. I remember wanting to do something like this, but I felt about the django.setup() thing that it wouldn’t be quite right, that it was more of a dirty hack and not the preferred solution. Otherwise, thanks for reassuring me that this is the way to go!

@KenWhitesell Thanks for reminding me of the need for some sort of supervisor process, I use docker so I think that will do the job.