Intermittent Data Truncation Issue with Django Endpoint

Using Django Mystery Errors
1

Hi everyone,

I’ve been facing an intermittent issue with one of my Django endpoints and I’m hoping someone here can help me figure out what’s going wrong.

The Problem

I have an endpoint that should return a large data dump (around 3000 names) when I send a POST request. Sometimes, it works perfectly and I get the full data. However, most times, it returns only a partial list and then abruptly stops, resulting in broken data. For example, the response might end with a string not fully closed or an incomplete list.

Backend Endpoint Code:

from django.views.decorators.csrf import csrf_exempt
from django.http import JsonResponse, HttpResponse
import json

@csrf_exempt
def grab_params(request):
    if request.method != 'POST':
        return HttpResponse("Invalid request method", status=400)
    
    data = json.loads(request.body)
    category = data['category']
    
    # Query the dataframe for the data
    current_df = search()
    
    # Extract list of unique parameters and sort appropriately
    params = current_df[category].unique().tolist()
    
    return JsonResponse(params, safe=False)

Frontend Call:

const response = await fetch(`${ip_address}/backend/param-list/`, {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
    },
    body: JSON.stringify({ category }),
});
const data = await response.json();

Example of Stuck Data:

[
    "Name1",
    "Name2",
    "Name3",
    "Name4",
    "Name5",
    ...
    "Name6",
    "Nam

Running the Server:

python manage.py runserver

As you can see, the response is being cut off, and the string never gets closed, nor does the list.

What I’ve Tried So Far

  1. Checked server logs: No errors or warnings indicating what might be causing the issue.
  2. Increased timeout settings: Thought it might be a timeout issue, but no luck.
  3. Tested with different data sizes: Smaller data sets work fine consistently.
  4. Tried compressing the data: Didn’t make any difference.
  5. Originally built using Flask: Worked perfectly without any issues.
  6. Pagination is not an option: Due to specific requirements of the project.

Questions:

  1. Has anyone experienced similar issues with Django when returning large JSON responses?
  2. Could this be related to network issues or server configurations?
  3. Any suggestions on how to debug or resolve this problem?

Any help or insights would be greatly appreciated!

Thanks in advance!

Edit: This is running on an Apache server. Upon moving it onto a local system, it runs perfectly with no errors. Why would it cause problems on an Apache server but had no problems when it was a Flask API?