Swap values of model fields with unique constraint

Let’s say I have the following two models:

class Category(models.Model):
    # ...

class Item(models.Model):
    ordering = models.PositiveIntegerField()
    category = models.ForeignKey(Category)

    class Meta:
        ordering = ["category_id", "ordering"]
        constraints = [
            models.UniqueConstraint(
                fields=["category_id", "ordering"]
            )
        ]

I want to be able to swap the order of two Items within the same category.

However, I can’t simply do:

temp = item1.ordering
item1.ordering = item2.ordering
item1.save()
item2.ordering = temp
item2.save()

as that would trigger an integrity error because I’m violating the unique constraint.

The only two solutions I’m seeing are:

  • remove the unique constraint on
  • allow the field order to be NULL and temporarily set the it to that value during the swap.

I don’t particularly like either of those solutions as they both carry some serious drawbacks.

Is there another way to accomplish this? I’m thinking maybe some transaction shenaningans, but I haven’t been able to come up with anything yet.

I think using transaction.atomic is the cleanest approach.

with transaction.atomic():
    item1.ordering, item2.ordering = item2.ordering, item1.ordering
    item1.save()
    item2.save()

Did you try something similar to the above? If so, what were the results?

1 Like

You’ll want to set the deferrable option on the constraint. See Constraints reference | Django documentation | Django

2 Likes

I was not aware of that, good call!

Thank you for your help! It seems to be working. I’ll share some code showing how I used this in practice to implement the ability to drag&drop ordered items in my application, just in case this is useful to somebody in the future.

Let’s start from the abstract models I used:

class TrackFieldsMixin(models.Model):
    """
    Abstract model used to track changes to a model's fields before
    writing those changes to the db
    """

    TRACKED_FIELDS = []  # list of field names

    class Meta:
        abstract = True

    @classmethod
    def from_db(cls, db, field_names, values):
        instance = super().from_db(db, field_names, values)
        for fieldname in cls.TRACKED_FIELDS:
            setattr(instance, f"_old_{fieldname}", getattr(instance, fieldname))

        return instance


class OrderableModel(TrackFieldsMixin):
    ORDER_WITH_RESPECT_TO_FIELD = ""  # field name
    TRACKED_FIELDS = ["_ordering"]

    _ordering = models.PositiveIntegerField()

    class Meta:
        abstract = True

    def save(self, force_no_swap=False, *args, **kwargs):
        if self.pk is None:  # creating new instance
            # automatically populate the ordering field based on the ordering of siblings
            setattr(self, "_ordering", self.get_ordering_position())

        if self._old__ordering != self._ordering and not force_no_swap:
            target_ordering, self._ordering = self._ordering, self._old__ordering

            while target_ordering != self._ordering:
                to_be_swapped = (
                    self.get_next()
                    if target_ordering > self._ordering
                    else self.get_previous()
                )
                if to_be_swapped is not None:
                    self.swap_ordering_with(to_be_swapped)

        else:
            super().save(*args, **kwargs)

    def get_siblings(self):
        return type(self).objects.filter(
            **{
                self.ORDER_WITH_RESPECT_TO_FIELD: getattr(
                    self, self.ORDER_WITH_RESPECT_TO_FIELD
                )
            },
        )

    def get_adjacent(self, step):
        delta = step
        siblings = self.get_siblings()
        for _ in range(0, len(siblings)):
            try:
                return siblings.get(_ordering=self._ordering + delta)
            except type(self).DoesNotExist:
                delta += step

        return None

    def get_next(self):
        return self.get_adjacent(1)

    def get_previous(self):
        return self.get_adjacent(-1)

    def swap_ordering_with(self, other):
        if not isinstance(other, type(self)) or getattr(
            self, self.ORDER_WITH_RESPECT_TO_FIELD
        ) != getattr(other, self.ORDER_WITH_RESPECT_TO_FIELD):
            raise ValidationError("Cannot swap with " + str(other))

        with transaction.atomic():
            self._ordering, other._ordering = other._ordering, self._ordering
            other.save(force_no_swap=True)
            self.save(force_no_swap=True)

    def get_ordering_position(self):
        # filter to get parent
        filter_kwarg = {
            self.ORDER_WITH_RESPECT_TO_FIELD: getattr(
                self, self.ORDER_WITH_RESPECT_TO_FIELD
            )
        }

        # get all model instances that reference the same parent
        siblings = type(self).objects.filter(**filter_kwarg)

        max_ordering = siblings.aggregate(max_ordering=Max("_ordering"))["max_ordering"]
        return max_ordering + 1 if max_ordering is not None else 0

The TrackFieldsMixin, which @KenWhitesell helped me create in another past thread of mine, allows to track changes for some fields. This data can be used, for example, before saving a model instance to see if a specific field was modified.

OrderableModel is meant to be used with models that have a many-to-one relationship with another model where the ordering on the “many”-part matters.

For example, in my app I have a Question and a Choice model. I want to keep track of the ordering of choices for a question, and I want users to be able to change the order of choices in the UI via drag&drop.

What happens is the following: the user drops a choice to a new location on the UI, a PATCH request is issued to the server to change the _ordering of the choice, which inherits from OrderableModel.

The change to the field _ordering is detected using the TrackFieldsMixin-provided machinery, and the swapping happens.

If choice had _ordering=3 and the request sets it to 1, it will first be swapped with the one that has _ordering=2 (assuming no deletions happened, otherwise it’ll be the first one that has a lower value–or higher, if it was moved upwards), and then with the one that has value 1.

Lastly, the choice model looks like this:

class Choice(OrderableModel):
    question = models.ForeignKey(
        Question,
        related_name="choices",
        on_delete=models.CASCADE,
    )

    # other fields

    ORDER_WITH_RESPECT_TO_FIELD = "question"

    class Meta:
        ordering = ["question_id", "_ordering"]
        constraints = [
            models.UniqueConstraint(
                fields=["question_id", "_ordering"],
                name="same_question_unique_ordering",
                deferrable=models.Deferrable.DEFERRED,
            ),
        ]

How’s this look? I have only tested it in the shell and so far it seems to be working. I’ll write a few test cases. I have noticed that, if I try to edit the ordering field in the admin, it won’t work, probably because the deferred option is ignored during form validation.

I have yet to test this using the REST API of my app, so I still don’t quite know if it works with nested transactions and all. I have a little bit of fear that something might break if I use this with the real API, but I’ll test and see. Until then, if you have any suggestions, they’re very welcome.

Thank you!

I think this still may be susceptible to duplicates on the ordering if two are attempted to be added at the same time due to how get_ordering_position works. I think you’d need to wrap that call along with the insertion in a transaction.

I can’t speak to the potential race condition that @CodenameTim raises - but I’ll add that for the one case where we had to do something like that, we didn’t try to get anywhere near as precise.
We just used the jQuery UI sortable. On any change, we sent the entire sorted list to the server via AJAX and did a bulk update on the entire list. Granted, the lists were relatively small - 20 to 25 items, but the view worked out to about 12-15 lines of code and it worked extremely well for us.

I like this idea. Did you use bulk_update on the backend?

Does it matter that I’m using ATOMIC_REQUESTS = True in my settings? If I understand it correctly, that should prevent the issue.

That was gonna be my plan B, so to speak. But with this project, I made the conscious decision to use a pattern, with nested resources, where you don’t just send data for multiple entities at once and have the server do bulk operations. That’s because I have already experimented with that, and it does bring some issues that I really didn’t want to have to address.

For example, I’m also using this feature for another model that has multiple m2m fields and all that. Since I’m using DRF, sending the whole list each time would trigger the update method for the ListSerializer associated to the model (which isn’t provided by default and you have to write yourself), with a lot of questions associated to it: what happens if the payload contains an entity that didn’t exist until the time of the request? What if one that exists is left out? You could just create the former, delete the latter, and perform a complete update (PUT style) on the remaining ones, but my app is built around auto-saving and sending little PATCH requests to keep everything up to date without the user having to worry about saving, so that would break the pattern.

IIRC, yes, and with the updates all occurring in an atomic block. (But that was a long time ago in a galaxy far far away, and I wouldn’t trust my memory of that system any farther than I could throw a piano…)

1 Like

We didn’t require saves, either. Every change generated a POST for the update.
And while we didn’t use DRF at all at the time, the simplicity of the POST being generated would lead me to believe it would be unnecessary to use it in this case regardless.

The POST data was a simple list of the IDs of the objects in the display in the order they currently exist, with the sequence number being the position of the element in the list +1 (One-based, not zero-based).

1 Like