Truncated or oversized response headers received from daemon process 'django_app'

Using Django Deployment
1

Hello,

I have used Python-Django and created a website similar to route planning. I have developed this website locally and deployed it on a Linux server using Apache2 - mod_wsgi. Now when I was adding a new feature where I load a parking station(long, lat) on a mapbox map GUI. This parking stations data is a .obj file having a size of more than 1GB. When I am trying to load this whenever the website is refreshed or opened and I get the below error as the server is not responding and I am not able to load the park stations onto my map.


Truncated or oversized response headers received from daemon process 'django_app': 
[mpm_event:notice] [pid 292705:tid 139727821035392] AH00492: caught SIGWINCH, shutting down gracefully
Below is from the website source page:
Failed to load resource: the server responded with a status of 408 (Request Timeout)

I would like to know how to handle this large data as I followed the deployment using Django documentation.

Below is my .conf file

VirtualHost *:80>
        # The ServerName directive sets the request scheme, hostname and port that
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        #ServerName www.example.com

        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        <Directory "/var/www">
                AllowOverride None
                Require all granted
        </Directory>
        <Directory /home/h2fuel/routing_ems>
                Require all granted
        </Directory>
        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".
        #Include conf-available/serve-cgi-bin.conf
        Alias /static /home/h2fuel/routing_ems/route_planner/staticfiles

        <Directory /home/h2fuel/routing_ems/route_planner/staticfiles>
                Require all granted
        </Directory>

        <Directory /home/h2fuel/routing_ems/route_planner/route_planner>
                <Files wsgi.py>
                        Require all granted
                </Files>
        </Directory>

        WSGIScriptAlias / /home/h2fuel/routing_ems/route_planner/route_planner/wsgi.py
        WSGIDaemonProcess django_app python-path=/home/h2fuel/routing_ems/route_planner python-home=/home/h2fuel/routing_ems/route_planner/mipenv
        WSGIProcessGroup django_app
        WSGIApplicationGroup %{GLOBAL}
        LimitRequestBody 0
        Timeout 600
        LimitRequestLine 16384
        LimitRequestFieldSize 16384
</VirtualHost>

Thanks and Regards
Raghu Boppana

2

Welcome @raghuboppana !

Is this .obj file a static file, or data being generated by a Django view?

What is the url being issued to retrieve this data?

3

Hello @KenWhitesell ,

Thank you for your prompt response. The .obj file is stored in the root directory of my Project and I guess it is kind of static.

below is the function in my views.py that is called in my map.html as an async function for loading the park stations

def get_hgv_parking_locations(request):
    try:
        print("Request for parking locations received")
        file_name = '2024-06-14_Germany_v0.obj'
        file_path = os.path.join(project_root, file_name)
        print(f'file_path_parking:{file_path}')

        def file_iterator(file_path):
            """Generator that loads and yields objects from a large pickle file."""
            with open(file_path, 'rb') as f:
                try:
                    unpickler = pickle.Unpickler(f)
                    while True:
                        try:
                            # Attempt to load the next object from the pickle file
                            access_hgv = unpickler.load()

                            # Ensure it is a GeoDataFrame with 'Point' type geometries
                            hgv_locations = access_hgv[access_hgv.geom_type == 'Point']
                            hgv_coordinates = [(point.y, point.x) for point in hgv_locations.geometry]

                            # Yield JSON serialized chunk of HGV parking locations
                            yield json.dumps({'hgv_parking_locations': [
                                {'lat': lat, 'lng': lng} for lat, lng in hgv_coordinates
                            ]}) + '\n'  # Adding newline to separate chunks

                        except EOFError:
                            # End of file reached, exit loop
                            break
                        except Exception as e:
                            # Handle other exceptions during the unpickling process
                            logging.error(f"An error occurred while processing an object from the file: {e}")
                            logging.error(traceback.format_exc())
                            yield json.dumps({'error': str(e)})  # Send error as JSON if any exception occurs

                except Exception as e:
                    logging.error(f"An error occurred while initializing the unpickler: {e}")
                    logging.error(traceback.format_exc())
                    yield json.dumps({'error': str(e)})  # Send error as JSON if any exception occurs

        print("Parking stations data stream initiated")
        return StreamingHttpResponse(file_iterator(file_path), content_type='application/json')

    except Exception as e:
        logging.error(f"An error occurred: {e}")
        logging.error(traceback.format_exc())
        return JsonResponse({'error': 'An error occurred on the server'})

This is my urls.py

urlpatterns = [
    path('', views.create_map, name='create_map'),
    path('get_fuel_stations/', views.get_fuel_stations, name='get_fuel_stations'),
    path('get_hgv_parking/', views.get_hgv_parking_locations, name='get_hgv_parking_locations'),
]

Best Regards
Raghu