How we found 2 seconds of hidden latency in our Django app using eBPF

Using Django
1

We run Django + gunicorn + DRF in production. Our P95 API latency was around 5 seconds but OpenTelemetry could only explain about 3 seconds. The remaining 2 seconds were completely unaccounted for. No slow queries, no slow third-party calls. Just missing time.

We tried Datadog, we tried py-spy. Neither could explain the gap. So I built an eBPF profiler that hooks into CPython internals from kernel space.

Here’s what we found hiding underneath our Django app:

GIL contention from URL resolution. Django’s URL resolver was holding the GIL while resolving routes. We have a large urlconf and this was adding up on every request. Since our URLs are static, we added URL caching and cut GIL wait by 80%. The tool showed us both which thread was holding the GIL and which thread was waiting, with full Python stacks for both. No other profiler showed us this.

GC gen-2 pauses. Django loads a lot of objects at startup. Models, middleware, URL patterns, template engines. Gen-2 GC was walking 1.5M objects on each collection. Worst case was 1.27 seconds. We added gc.freeze() in our gunicorn post_fork hook to freeze startup objects. Worst-case GC pause dropped to 88ms.

Gunicorn handoff latency. When workers are busy with GIL contention or GC, new requests pile up in gunicorn’s queue. Our tool measures exact handoff time. Before fixes: requests waited seconds in queue. After fixes: single-digit milliseconds.

Off-CPU from Kafka. One of our DRF views was pushing to Kafka synchronously. The thread was blocking for 130ms per request. Completely invisible in OpenTelemetry because it wasn’t instrumented. Our tool showed the full stack: views.py:post > sessions.py:post > socket.py:readinto. We moved it to async and the off-CPU time disappeared.

The tool uses eBPF uprobes on CPython’s take_gil/drop_gil and gc_collect_main. No code changes to your Django app. Just point it at your gunicorn PIDs and run.

CPython 3.11 only for now. Open source under Apache 2.0.

GitHub: GitHub - deepanshu406/python-ebpf-profiling: eBPF-powered Python profiling toolkit CPU profiling, GC pauses, GIL contention, off-CPU analysis, network health · GitHub

Has anyone else run into unexplained latency gaps in their Django app? Curious what tools you use to debug this.

2

Thanks for sharing this. The GIL finding is the one I’d like to understand better. In standard GIL-enabled CPython, Django’s resolver runs while holding the GIL; that’s different from showing that other threads are blocked waiting for it. Which Gunicorn worker class and thread count were you using? With sync workers, each worker is a separate process, so one worker’s GIL contention won’t block another worker.

Django compiles URL patterns and checks them in order for each request. Did your optimization cache path-to-ResolverMatch results? If so, how do you handle request-specific URLconfs or locale-dependent patterns?

I’m also curious about the gc.freeze() placement. Python documents freezing just before fork(), then enabling GC in each child. How does the post_fork hook behave with Gunicorn’s preload_app enabled versus disabled?

A benchmark with the worker configuration, Python build, profiler overhead, and definition of “handoff time” would make the results easier to evaluate.

References: Python gc.freeze() · Django URL dispatcher · Gunicorn worker model · Gunicorn preload_app setting