Usage of reprlib instead of pprint in exception reporting

Django Internals
1

Context: #34746 (High CPU/memory consumption when a 5XX is raised with large local variables) – Django (djangoproject.com)

Summary: The exception reports in Django use the pprint to pretty print variables in the stack frame of the exception. The pprint function uses a lot of memory and time to render the entire object and trim it to the desired length later. This becomes significant if the variable is of huge size.

I’m planning to prevent this by using the reprlib like the previous contributors to the ticket and finish it.

First I wanted to implement a kind of size check and leave the variables that are not oversized to pprint itself so that the experience is not altered much, but seems like there is no fast and accurate way to measure large variables’ memory in Python. It’s either inaccurate or becomes really slow to deeply evaluate a variable’s memory usage.

Thus, it’s best to completely switch to reprlib. We can to try add indentations and limits within the instance constructor so that we get our desired result.

I was planning to use ‘4096’ the current limit for all data types and indentation as 4.
Please post any feedback on the same.

Reference for reprlib configuration:

2

Hi! Thank you for working on this gnarly ticket.

Yes, it’s hard to figure out a variable’s size. C-level objects can obscure their memory usage further.

Completely switching to reprlib sounds fine by me if the indentation keeps things fairly readable. Providing some before/after examples in your PR would help with reviewing what this change will entail.

1 Like
3

Hi keerthivasansa, I too had been looking into this ticket here. Adam mentioned about you, and I see you have already raised a PR. Let me know if you need anything, would be happy to collaborate!