I am trying to add multiple entries to populate an SQLite3 database from a POST request.
In total, there are about 2000 entries, but every time I execute the POST request, the server shuts down (without printing any error in the console or in the browser).
The total size of the entries does not exceed 200KB, but the server always crashes after processing 334 entries (if I upload one by one), or 300 entries (if I upload in batches of 100).
I have tried modifying “DATA_UPLOAD_MAX_NUMBER_FIELDS” (each model entry has 20 parameters) to 10.000 to be sure, “DATA_UPLOAD_MAX_MEMORY_SIZE” and "FILE_UPLOAD_MAX_MEMORY_SIZE to 25mb, but it doesn’t seem to have any effect. It also doesn’t give me any error that would help me understand the problem.
This is my code structure:
views.py (it loads 334 entries before shutting down)
def settings(request):
if request.method == "POST" and "update_report" in request.POST:
Item.objects.all().delete()
from xxx import items
try:
for item in items:
Item.objects.create(
propertyA = item.get('A'),
propertyB = item.get('B'),
propertyC = item.get('C'))
except IntegrityError as e:
print(f"Integrity error: {e}")
template = loader.get_template("xxxx\yyyy.html")
context = {some context}
return HttpResponse(template.render(context, request))
views.py (this other attemp only upload 300 items before shutting down)
def settings(request):
if request.method == "POST" and "update_report" in request.POST:
Item.objects.all().delete()
from xxx import items
items_list = [Item(propertyA = item.get('A'),
propertyB = item.get('B'),
propertyC = item.get('C'))
for item in items]
chunk_size = 100
try:
for i in range(0, len(items_list), chunk_size):
Item.objects.bulk_create(items_list[i:i+chunk_size])
except IntegrityError as e:
print(f"Integrity error: {e}")
template = loader.get_template("xxxx\yyyy.html")
context = {some context}
return HttpResponse(template.render(context, request))
html code
....
<form method="post">
{% csrf_token %}
<button type="submit" name="update_report">Update report>/button>
<form>
....
settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': 'debug.log',
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
debug.log (last lines)
(0.000) INSERT INTO "my_app_item" ("propertyA", "propertyB", "propertyC") VALUES ("AAA", "AAA", "AAA")....
(0.000) INSERT INTO "my_app_item" ("propertyA", "propertyB", "propertyC") VALUES ("BBB", "AAA", "AAA")....
(0.000) INSERT INTO "my_app_item" ("propertyA", "propertyB", "propertyC") VALUES ("CCC", "CCC", "CCC")....
thank you