incremental bulk deletion

Using Django Using the ORM
2

:waving_hand: hello there!

I think you might have misread the documentation

The delete() method does a bulk delete and does not call any delete() methods on your models. It does, however, emit the pre_delete and post_delete signals for all deleted objects (including cascaded deletions).

On the topic of deletion itself.

I’m pretty sure this will not do what you want and will be relatively expensive given how poorly LIMIT / OFFSET behaves on large data set.

First it won’t do what you expect (deleting all the rows) because as you page through the results (assuming you have defined a total order on the results) you will have deleted the page before already.

Say that you have 10,000 rows matching is_expired ordered by their primary key then the first iteration would find the first page_range rows and delete them. The thing is that when the first iteration runs the second page has become the first the third page has become the second so you’ll skip over the first. In other words this approach will only delete half the records as all initially even pages will be skipped.

A better approach would be keep fetching records until they exist, letting the database figure out what’s the most efficient way to do so. Something like

chunk_size = 1000
queryset = (
    Model.objects.filter(
        is_expired=True,
    )
    .order_by()
    .values_list("pk", flat=True)
)
while True:
    deleted, _ = Model.objects.filter(
        pk__in=queryset[:chunk_size]
    ).delete()
    if not deleted:
        break

this will perform queries of the form

SELECT * FROM model WHERE id IN (
    SELECT id FROM model WHERE is_expired LIMIT ?
)
---
DELETE FROM model WHERE id IN ?

for each iteration if you have any pre_delete or post_delete signal registered or have other models pointing at YouModel with ForeignKeys or

DELETE FROM model WHERE id IN (
    SELECT id FROM model WHERE is_expired LIMIT ?
)

otherwise.