Hello !
I’m working on an application. In this app i use python multiprocessing and django for the orm.
This app is made for running in docker container.
Sorry in advance for all the code block but it is important to understand the problem.
I will give you some context to understand the problem i’ve make an verry little version to better comprehension.
my model:
from django.db import models
class User(models.Model):
id = models.AutoField(primary_key=True)
name = models.CharField(max_length=255, blank=True, null=True)
class DataType(models.Model):
value = models.CharField(unique=True, max_length=62)
label = models.CharField(max_length=32, null=True)
class UserData(models.Model):
data_type = models.ForeignKey(DataType, models.DO_NOTHING)
user = models.ForeignKey(User, on_delete=models.deletion.DO_NOTHING)
my program:
from db.models import *
import multiprocessing
def process_data(data, nb):
print(f"pid: {nb}, data pk: {data.pk}")
print(f"pid: {nb}, data type id: {data.data_type_id}")
print(f"pid: {nb}, data type: {data.data_type}")
def generate():
task_list = []
data_list = [UserData.objects.get(id=x) for x in (10, 11)]
nb = 0
for data in data_list:
print(f"data user id: {data.user.id}")
task = multiprocessing.Process(target=process_data, args=(data, nb))
task_list.append(task)
task.start()
nb += 1
for task in task_list:
task.join()
print("End of tasks")
if __name__ == '__main__':
generate()
on my database i’ve this:
MariaDB [bb]> select * from db_user;
+------+--------+
| id | name |
+------+--------+
| 1001 | Pierre |
| 1002 | Paul |
+------+--------+
2 rows in set (0.002 sec)
MariaDB [bb]> select * from db_datatype;
+----+-------+--------+
| id | value | label |
+----+-------+--------+
| 1 | data1 | Data 1 |
| 2 | data2 | Data 2 |
| 3 | data3 | Data 3 |
+----+-------+--------+
3 rows in set (0.001 sec)
MariaDB [bb]> select * from db_userdata;
+----+--------------+---------+
| id | data_type_id | user_id |
+----+--------------+---------+
| 10 | 2 | 1001 |
| 11 | 2 | 1002 |
+----+--------------+---------+
2 rows in set (0.002 sec)
Now my problem: When i run directly on my (windows) computer the program, i have this output:
data user id: 1001
data user id: 1002
pid: 0, data pk: 10
pid: 0, data type id: 2
pid: 1, data pk: 11
pid: 1, data type id: 2
pid: 0, data type: DataType object (2)
pid: 1, data type: DataType object (2)
End of tasks
No problem here.
The problem appear when i run this program in docker container. my output is like here:
2024-04-03 16:43:12 data user id: 1001
2024-04-03 16:43:12 data user id: 2
2024-04-03 16:43:12 Process Process-1:
2024-04-03 16:43:12 Traceback (most recent call last):
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 235, in __get__
2024-04-03 16:43:12 rel_obj = self.field.get_cached_value(instance)
2024-04-03 16:43:12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
2024-04-03 16:43:12 return instance._state.fields_cache[cache_name]
2024-04-03 16:43:12 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
2024-04-03 16:43:12 KeyError: 'data_type'
2024-04-03 16:43:12
2024-04-03 16:43:12 During handling of the above exception, another exception occurred:
2024-04-03 16:43:12
2024-04-03 16:43:12 Traceback (most recent call last):
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 122, in __iter__
2024-04-03 16:43:12 obj = model_cls.from_db(
2024-04-03 16:43:12 ^^^^^^^^^^^^^^^^^^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 578, in from_db
2024-04-03 16:43:12 values = [
2024-04-03 16:43:12 ^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 579, in <listcomp>
2024-04-03 16:43:12 next(values_iter) if f.attname in field_names else DEFERRED
2024-04-03 16:43:12 ^^^^^^^^^^^^^^^^^
2024-04-03 16:43:12 StopIteration
2024-04-03 16:43:12
2024-04-03 16:43:12 The above exception was the direct cause of the following exception:
2024-04-03 16:43:12
2024-04-03 16:43:12 Traceback (most recent call last):
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2024-04-03 16:43:12 self.run()
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
2024-04-03 16:43:12 self._target(*self._args, **self._kwargs)
2024-04-03 16:43:12 File "/app/main.py", line 8, in process_data
2024-04-03 16:43:12 print(f"pid: {nb}, data type: {data.data_type}")
2024-04-03 16:43:12 ^^^^^^^^^^^^^^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 253, in __get__
2024-04-03 16:43:12 rel_obj = self.get_object(instance)
2024-04-03 16:43:12 ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 216, in get_object
2024-04-03 16:43:12 return qs.get(self.field.get_reverse_related_filter(instance))
2024-04-03 16:43:12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 643, in get
2024-04-03 16:43:12 num = len(clone)
2024-04-03 16:43:12 ^^^^^^^^^^
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 380, in __len__
2024-04-03 16:43:12 self._fetch_all()
2024-04-03 16:43:12 File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1926, in _fetch_all
2024-04-03 16:43:12 self._result_cache = list(self._iterable_class(self))
2024-04-03 16:43:12 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-03 16:43:12 RuntimeError: generator raised StopIteration
2024-04-03 16:43:12 pid: 0, data pk: 10
2024-04-03 16:43:12 pid: 0, data type id: 2
2024-04-03 16:43:12 pid: 1, data pk: 11
2024-04-03 16:43:12 pid: 1, data type id: 2
2024-04-03 16:43:12 pid: 1, data type: DataType object (2)
2024-04-03 16:43:12 pid: 1, end process
2024-04-03 16:43:12 End of tasks
On this output, we can see on the firsts lines an problem, on the second line, the id of the user in totally incorrect, this id doesn’t exists in database.
if i re run the program, sometimes i have the same error, sometimes it’s an other field it’s incorrect.
It see like an memory leak issue but i’m not really sure and i dont’ have found an solution to this.
If you have needs to view my dockerfile, docker-compose, or other file, don’t hesitate to ask me
Thanks for persons take time to read this <3