`Process` or `Thread` problem deploying Django app with uWSGI

Using Django Deployment
1

Hello Django devs. We are currently facing weird threading problems deploying a Django app with uWSGI. Any help would be highly appreciated :blush:!

The problem is on the background gRPC process we spawn using multiprocessing.Process. It possesses a multiprocessing.connection.Connection and uses it to send data to the Django process to save them in the database.

2

The above implementation works fine using runserver :+1:, but gets stuck when we try deploying using uWSGI :-1:. We identified that the background process gets stuck when it tries to send data using that Connection. Strangely, making any HTTP request to our server “un-stucks” the background process :person_shrugging:.

Switching to threading.Thread instead of Process & Connection doesn’t work, either :cry:. The background process gets stuck right at the beginning using uWSGI. Although, it works when using runserver :person_shrugging:.

I get that uWSGI disables the GIL or does some hacks that interfere with threading and multiprocessing. But, we don’t know how to work around that, so we would like help from you guys :blush:.

We also would like to avoid having to employ a queue system that involves brokers and extra databases for simplicity.

Thanks in advance :rose:!

3

Yep.

Don’t try to start or manage background processes from within your wsgi process.

That’s the reason why tools such as Celery and the worker processes for Channels exist.

Search this forum for “Background tasks” to find the threads that explain all the reasons why this is an extremely bad idea.

4

Thank you for your insight. Though, that is really unfortunate and disappointing. I had looked through many popular worker processors, but found them overly complicated and their functionality not that valuable to us, so I decided to simply use multiprocessing.

I guess you are right. WSGI is probably the big reason why these libraries are needed.

I appreciate your clear take.