I’ve been learning more about logging in Django/Python, and I’m stuck on the best way to approach log aggregation in production. From what I understand, if I use something like logtail-python (or any handler that ships logs directly to an external service), those logs are sent synchronously in the request cycle.
That raises a few concerns:
- Does this add noticeable overhead to the request/response cycle?
- What happens if the logging service is slow or unavailable, does the handler block until timeout, retry, or just drop logs? (I haven’t found anything clear in the
logtail-python docs)
I know the standard setup (file, console, rotating file handler) is simple and fast, but then you lose centralized aggregation unless you run a log agent (which I can’t do on Render free tier).
So my questions are:
- In practice, how do Django apps in production usually handle log aggregation?
- Is it more common to send logs directly from the app, or to rely on stdout + a log agent/service that ships asynchronously?
- For those who’ve run Django on platforms like Render (with free tier constraints), how did you set things up?
Would love to hear how others are handling this.
Welcome @enamyaovi !
For the most part, we use the rotating file handler. It performs well-enough for what we rely upon it to do.
In some cases we have used either rsyslog to forward logs and/or the nxlog agent for centralized collection. In both cases, they forward logs over UDP which is a “best effort” transmission. However, on the internal network, the success rate is well above 99%. (Obviously, if you absolutely need 100% complete logs, then this is not a viable solution.)
This is going to depend upon the nature of the logs being collected and what you’re planning to do with them.
I don’t even consider such a deployment. From my perspective, the hidden costs of a “free tier” far exceed the price of a more flexible environment given what a common deployment looks like for me.
If you’ve got a defined business need for these logs to (a) not add any overhead and (b) have a reliability exceeding ~ 99.999%, then you’re going to want to pay for an environment to support this.
If performance and reliability are that important, I’d consider using the QueueHandler to dispach the log messages to an external process.
I’d recommend divorcing log aggregation from your application entirely and instead just dump logs to stdout. That’s about the fastest you’re ever going to get outside of doing something stragely clever like caching the log messages in-memory and emitting them after the response via middleware (ew). That’d be kinda dangerous since you won’t get any logs in the event of a failure or a service being killed.
Once you’re logging to stdout, you can set up a separate service to consume those logs and aggregate them. In Kubernetes-land, this is pretty standard practice and you can get off-the-shelf solutions like Grafana or the ELK stack to help you with it all, but if you’re foregoing K8s in favour of something like Render, I’m not sure how that’d work honestly. If that setup will allow you to stream to stdout/console and let you hook another service up to consume that stream, then I suppose the effect would be the same.
thanks a lot @KenWhitesell. I am new to django/web-dev in general. Have only worked on personal projects so far after my bootcamp. Your answers clarified the questions on my mind, I was curious as to what was done in practice and the reason I still use free tier is because I am still learning the ropes. I’ll look into the QueueHandler’s to see what I can find. Thanks once again
mmm okay, thank you @danielquinn, I guess the solution is already out there and I am making my life more difficult by using Render’s free tier as is. I’ll look into Grafana and the ELK stack for when I have not-beginner projects I have worked on that will need the log aggregation. I was curious as to how it’s really done in practice.