Django / Docker app out of memory after switching to Gunicorn

Using Django Mystery Errors
1

I get the following error when trying to run a ML/AI app in Django/Docker. I started getting the error after switching to Gunicorn. I understand it is due to memory allocation limitations, but I am not sure how to fix it.

 [2024-06-18 08:56:09 -0500] [19] [INFO] Worker exiting (pid: 19)
web-1  | [2024-06-18 13:56:10 +0000] [1] [ERROR] Worker (pid:19) was sent SIGKILL! Perhaps out of memory?
web-1  | [2024-06-18 13:56:10 +0000] [34] [INFO] Booting worker with pid: 34
web-1  | /usr/local/lib/python3.11/site-packages/whisper/__init__.py:63: UserWarning: /root/.cache/whisper/tiny.pt exists, but the SHA256 checksum does not match; re-downloading the file
web-1  |   warnings.warn(
 78%|█████████████████████████████        | 56.6M/72.1M [00:24<00:05, 2.74MiB/s][2024-06-18 13:57:18 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:34)
 79%|█████████████████████████████        | 56.7M/72.1M [00:24<00:06, 2.42MiB/s]
web-1  | [2024-06-18 08:57:18 -0500] [34] [INFO] Worker exiting (pid: 34)
web-1  | [2024-06-18 13:57:19 +0000] [1] [ERROR] Worker (pid:34) exited with code 1
web-1  | [2024-06-18 13:57:19 +0000] [1] [ERROR] Worker (pid:34) exited with code 1.
web-1  | [2024-06-18 13:57:19 +0000] [45] [INFO] Booting worker with pid: 45
web-1  | /usr/local/lib/python3.11/site-packages/whisper/__init__.py:63: UserWarning: /root/.cache/whisper/tiny.pt exists, but the SHA256 checksum does not match; re-downloading the file
web-1  |   warnings.warn(
 72%|██████████████████████████▊          | 52.2M/72.1M [00:24<00:10, 2.03MiB/s][2024-06-18 14:02:49 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:45)
 73%|██████████████████████████▉          | 52.4M/72.1M [00:24<00:09, 2.22MiB/s]
web-1  | [2024-06-18 09:02:49 -0500] [45] [INFO] Worker exiting (pid: 45)
web-1  | [2024-06-18 14:02:49 +0000] [1] [ERROR] Worker (pid:45) exited with code 1
web-1  | [2024-06-18 14:02:49 +0000] [1] [ERROR] Worker (pid:45) exited with code 1.
web-1  | [2024-06-18 14:02:49 +0000] [56] [INFO] Booting worker with pid: 56
web-1  | /usr/local/lib/python3.11/site-packages/whisper/__init__.py:63: UserWarning: /root/.cache/whisper/tiny.pt exists, but the SHA256 checksum does not match; re-downloading the file
web-1  |   warnings.warn(
100%|█████████████████████████████████████| 72.1M/72.1M [00:22<00:00, 3.39MiB/s]
web-1  | [2024-06-18 14:20:30 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:56)
web-1  | [2024-06-18 09:20:30 -0500] [56] [INFO] Worker exiting (pid: 56)
web-1  | [2024-06-18 14:20:31 +0000] [1] [ERROR] Worker (pid:56) exited with code 1
web-1  | [2024-06-18 14:20:31 +0000] [1] [ERROR] Worker (pid:56) exited with code 1.
web-1  | [2024-06-18 14:20:31 +0000] [79] [INFO] Booting worker with pid: 79

I ran docker stats to analyse memory usage. This is the output:

CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.06%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.00%     19.68MiB / 7.387GiB   0.26%     20.8kB / 18.1kB   0B / 0B     7
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.06%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.00%     19.68MiB / 7.387GiB   0.26%     20.8kB / 18.1kB   0B / 0B     7
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.03%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.03%     19.68MiB / 7.387GiB   0.26%     20.8kB / 18.1kB   0B / 0B     7
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.03%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.03%     19.68MiB / 7.387GiB   0.26%     20.8kB / 18.1kB   0B / 0B     7
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.02%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.01%     19.68MiB / 7.387GiB   0.26%     20.8kB / 18.1kB   0B / 0B     7
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.02%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.01%     19.68MiB / 7.387GiB   0.26%     20.8kB / 18.1kB   0B / 0B     7
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.93%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.72%     21.15MiB / 7.387GiB   0.28%     21.7kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   0.93%     571.5MiB / 7.387GiB   7.56%     278MB / 9.01MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.72%     21.15MiB / 7.387GiB   0.28%     21.7kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   3.78%     571.5MiB / 7.387GiB   7.56%     278MB / 9.02MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   3.78%     571.5MiB / 7.387GiB   7.56%     278MB / 9.02MB    0B / 0B     13
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   106.80%   681.8MiB / 7.387GiB   9.01%     279MB / 9.04MB    0B / 0B     14
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   106.80%   681.8MiB / 7.387GiB   9.01%     279MB / 9.04MB    0B / 0B     14
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   11.82%    692.5MiB / 7.387GiB   9.15%     279MB / 9.05MB    0B / 0B     14
0ac4bc893e43   djangoprojects-db-1    0.01%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   11.82%    692.5MiB / 7.387GiB   9.15%     279MB / 9.05MB    0B / 0B     14
0ac4bc893e43   djangoprojects-db-1    0.01%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   9.66%     697.1MiB / 7.387GiB   9.22%     280MB / 9.06MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   9.66%     697.1MiB / 7.387GiB   9.22%     280MB / 9.06MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   5.36%     697MiB / 7.387GiB     9.21%     280MB / 9.06MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   5.36%     697MiB / 7.387GiB     9.21%     280MB / 9.06MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   5.86%     697MiB / 7.387GiB     9.21%     280MB / 9.07MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   5.86%     697MiB / 7.387GiB     9.21%     280MB / 9.07MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   6.62%     697.1MiB / 7.387GiB   9.22%     281MB / 9.08MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   6.62%     697.1MiB / 7.387GiB   9.22%     281MB / 9.08MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.00%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   7.41%     697MiB / 7.387GiB     9.21%     281MB / 9.09MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.01%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   7.41%     697MiB / 7.387GiB     9.21%     281MB / 9.09MB    0B / 0B     22
0ac4bc893e43   djangoprojects-db-1    0.01%     21.15MiB / 7.387GiB   0.28%     21.8kB / 19.5kB   0B / 0B     8
CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
dab126e56daf   djangoprojects-web-1   9.39%     680.2MiB / 7.387GiB   8.99%     282MB / 9.1MB     0B / 0B     13

I can see that in the Net I/O section, memory usage is greater than the limit. However, I’m not sure how to make the necessary changes. I also read that here that simply increasing the Gunicorn timeout setting is only a band-aid solution. I’m completely stumped. Any help is much appreciated.

2

The “WORKER TIMEOUT” message tends to mean it took too long. Gunicorn defaults to a maximum of 30 seconds per request, but you can change that. It’s a band-aid solution because usually you don’t want a user to wait that long for a response. If a request can take longer than that, you should try and optimise the work it’s doing, or move the work out of the request-response cycle and into a background worker.

From the docker stats output, it doesn’t look like memory goes too high, although it could be from something else on the system, and the kernel chose to kill Django. I’d try and solve the timeout first rather than going there too much - “Perhaps out of memory” is intentionally a question.