I understand that Django revolves around the request/response cycle, so please forgive me if this seems like a silly question.
I am making the assumption that TCP connections to databases and services like memcached are not set up and torn down on each request/response. I assume that when we execute “manage.py runserver” certain classes/processes are initiated that are indeed outside the request/response cycle.
I have written a lazy connection class that pulls data from a RabbitMQ server. Initiating it in a view to retrieve the data works fine but it means that the TCP connection to RabbitMQ gets set up and torn down on every request.
I am looking for a way to instantiate this lazy class in a way more like the way Django sets up its database or memcache classes so that my RabbitMQ data can be used in views during the request/response cycle but without setting up anew TCP connection every time.
I hope what I am asking makes sense. Kind regards Adrian.
Welcome @AdrianFretwell !
These are not valid assumptions.
Database connections aren’t “permanent”, and can get made at the beginning of a request.
In the docs for Persistent Connections:
Django opens a connection to the database when it first makes a database query.
and, regarding the CONN_MAX_AGE parameter:
The default value is 0 , preserving the historical behavior of closing the database connection at the end of each request.
(This is one of the reasons why it is often recommended to use something like pgpool.)
As far as libmemcached is concerned, the library manages its own connections. Django - or even pylibmc - do not appear to be directly involved with that. From what I can see, the work is done down in the guts of the libmemcached C++ core object.
So yes, your question makes sense - but no, Django doesn’t do it that way.
You can implement something to do this yourself - you could create a RabbitMQ connection pooling feature. See Connection pool implementation and the source code for some ideas.
(Keep in mind that a production Django deployment may be running in a multithreaded / multiprocessing environment, and globablly sharing connections may not be the best idea.)
But I would probably want to make sure that this is a real issue before expending the effort of remediating it. Have you measured the additional latency created by the connection / disconnection relative to the overall duration of the view?
(It’s unfortunate that RabbitMQ doesn’t support unix domain socket connections.)
Addendum:
I also need to point out here that regardless of what runserver may do during development should not be construed as what happens during a real production deployment using something like gunicorn, uwsgi, or one of the alternatives. The development tool runserver should not be used as your production target.
Ken, thank you very much, your response is very helpful. I believe I understand a little better now.
I do use uwsgi in production.
I also use Paramiko for getting information from local servers using SSH in some views, so again I was thinking about the performance of creating and destroying those connections on every request as well as the RabbitMQ connections. Maybe I am just over-thinking this and as you say there may be very little performance advantage in trying to fix this when it probably does not need fixing.
Thank you again for your help. Kind regards, Adrian.