Error 500 during burpsuite scan only, otherwise runs well

Using Django Mystery Errors
1

Hello Django community

I have an issue here I am not sure how to approach.

I run a Django website in production, everything runs well for a while, thousands of hits, all good.

Then during weekends, there is an authorized third party that use something like burpsuite to scan my site. It scans for a while, my site works until eventually, it crash, generating web 500 errors. Touching wsgi.py restart the site and all is good until next scan.

The cause of the error 500 is psycopg not being present as a module:

[Sat Mar 30 08:49:28.944928 2024] [wsgi:error] [pid 739152] [remote this-ip] mod_wsgi (pid=739152): Failed to exec Python script file mysite/www/wsgi.py'.
[Sat Mar 30 08:49:28.944993 2024] [wsgi:error] [pid 739152] [remote this-ip] mod_wsgi (pid=739152): Exception occurred processing WSGI script mysite/www/wsgi.py'.
[Sat Mar 30 08:49:28.966613 2024] [wsgi:error] [pid 739152] [remote this-ip] Traceback (most recent call last):
[Sat Mar 30 08:49:28.966660 2024] [wsgi:error] [pid 739152] [remote this-ip]   File "mysite/venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 25, in <module>
[Sat Mar 30 08:49:28.966665 2024] [wsgi:error] [pid 739152] [remote this-ip]     import psycopg as Database
[Sat Mar 30 08:49:28.966683 2024] [wsgi:error] [pid 739152] [remote this-ip] ModuleNotFoundError: No module named 'psycopg'

if I look at the error in base.py, line 25:

try:
    try:
        import psycopg as Database # line 25
    except ImportError:
        import psycopg2 as Database
except ImportError:
    raise ImproperlyConfigured("Error loading psycopg2 or psycopg module")

So this looks for either psycopg or psycopg2; we use psycopg2; pip list show that is is indeed installed:

psycopg2                 2.9.6

yet at line 25 when trying to import psycopg and is not trying the except: for psycopg2

also I have many errors related to psycopg during the same event until I touch wsgi.py, some of these errors are reaching line 27 of base.py with an output like this

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 or psycopg module

I am not sure how to approach this, since this happens when there is a scan, and else works well the rest of the time. It feels like if the venv is ‘shutting down somewhat’ and another default system venv may not have the modules installed ? But why is it shutting down in the first place?

As a remark, about the queries in the apache log that are apparently breaking the site : if I try to replay the queries myself using curl, with the referrer, user-agent and all this, it will be handled well (redirection, page not found and no error 500)

As it is sporadic (weekly) it is difficult to investigate - is there any tricks with the logging that could improve the details levels (presently at DEBUG level) or other way to monitor the health of the venv maybe?

any tips would greatly help, thank you

2

How are you running your server process?

3

Hello Ken,

thank you for your reply!

yes good point, here it is:

I use an up to date apache2 on a recent and updated debian with mod_wsgi: follows the relevant apache configuration for wsgi

    WSGIProcessGroup mysite
    WSGIDaemonProcess mysite python-path=/mypath/site-root/mysite python-home=/mypath/site-root/mysite/venv/ display-name=%{GROUP}
    WSGIScriptAlias / /mypath/site-root/mysite/www/wsgi.py process-group=mysite

python version is the one shipped with the distro, as well as the corresponding apache/mod_wsgi (python 3.9 for debian 11)

first line of the daily apache log:

[Sat Mar 30 00:00:05 2024] [mpm_prefork:notice] [pid xxx] AH00163: Apache/2.4.56 (Debian) OpenSSL/1.1.1w mod_wsgi/4.7.1 Python/3.9 configured – resuming normal operations

4

Yes, setting the Apache log to Debug may yield some useful information. (See Debugging Techniques — mod_wsgi 5.0.0 documentation) Increasing the log level within Django may be helpful as well.

You may also want to find a way to monitor / check the connections to the database. Do you have something that might be interfering with the connection to it?

You’ll also want to look for patterns as to when this occurs - are these at about the same time of day every week? If so, look for infrastructure-related issues - updates being applied, processes being restarted, etc. Check syslog and other logs as to what may be happening during that time.

Being difficult to investigate is putting it mildly.

You mention this as possibly being related to a security test (you specifically mentioned burpsuite). You might want to try and correlate these events with certain types of tests. If there’s a way to segment certain types of tests, or avoid running some tests on particular weekends, you may be able to identify a test that is causing your site to fail - that, in itself, would have significant value.

If you had the ability to set up a test system and run your security tests against it to see if you can make your test system fail, that would also be useful.

1 Like
5

Hello Ken,

thank you for taking this time with your insight. Few great pointers there! Really appreciate.

I will take the time to implement improved logging levels for the different pieces, wherever I can. Also I will try to correlate events further.

Doing it in a lab (in a self contained network and try to make it fail) was also on the table, will go for it too.

I will update when I will have development on the matter

Cheers,

flax