ERR_CONNECTION_CLOSED after long operation

Using Django Mystery Errors
1

Hello there,

In my django project I perform some Machine Learning operations using GBOML and my algorithm can take up to 800-1000 seconds to perform.

I’m using the concept of Process to allow other users to use the application while the algorithm is running.

The process calls another method to perform all the operations, but it looks like this:

 try:
            terms_type = request.POST.get('terms')
            discount_rate = float(request.POST.get('discount_rate'))
            project_lifetime = int(request.POST.get('lifetime_project'))
            proc = Process(
                target=interface.execute,
                kwargs={
                    'request' : request,
                    'project_lifetime' : project_lifetime,
                    'discount_rate' : discount_rate,
                    'terms_type' : terms_type,
                    'scenario' : scenario,
                }
            )
            proc.run()
            scenario.last_run_date = datetime.now()
            scenario.save()
        except InfeasibleOptimizationException as e:
            logger.exception(f'InfeasibleOptimizationException raised in scenario {scenario.id}')
            messages.error(self.request, e.message)
            return HttpResponseRedirect(reverse('optimization:scenario-second', kwargs={'scenario_id': scenario_id}))
        except ValueError as e:
            logger.exception(f'ValueError raised in scenario {scenario.id}')
            messages.error(self.request, _('An error occured in your optimization. Please review your params'))
            return HttpResponseRedirect(reverse('optimization:scenario-second', kwargs={'scenario_id': scenario_id}))
        except Exception as e:
            logger.exception(f'Exception raised in scenario {scenario.id}')
            messages.error(self.request, _('An error occured in your optimization. Please review your params'))
            return HttpResponseRedirect(reverse('optimization:scenario-second', kwargs={'scenario_id': scenario_id}))

        return HttpResponseRedirect(reverse('optimization:results', kwargs={'scenario_id': scenario_id}))

The thing is, when my code takes around 300-400 seconds, everything’s good, at the end I am returned to the optimization:results route. But when the operation takes up to 800-1000 seconds (around those numbers), the process ends as I can see the results files created in my media folders, but I am never redirected to the optimization:results route. Instead, my browser shows an ERR_CONNECTION_CLOSED error.
The user needs to refresh the tab, go back to the home page and select the specific scenario to see the results (and it works). But it should automatically redirect to the results page and not show the ERR_CONNECTION_CLOSED error…

What should I change ?

Appreciate your help !

2

Welcome @david-silva987 !

You’re encountering one of (possibly) 4 or 5 different timeout situations.

Effectively, there’s nothing you can do. Unless you’re working with a completely internal and fully controlled network environment, at some point, these connections will time out - and will be disconnected by equipment between you and the browser that is making the request.

Side note: On a different issue regarding the stability and reliability of your system, you should never start processes from within your Django process. For more on this, see my responses at Django thread for opc ua/ modbus connection - #2 by KenWhitesell and I'm new to WebDev, explain to me why or why it wouldn't be OK for all users to access a global Lark() parser object? - #7 by derek-olson, and the messages they link.

As a general pattern, you should be using Celery (or something like it) to run these types of long-running tasks. They don’t belong / are not properly suited for being run within your Django context.