Django REST - Difficult nested relations to handle in update and create methods

Using Django
1

I’m trying to refactor my DRF app where I have the following models:

class Exam(models.Model):
    name = models.TextField()
    # ...

class Category(models.Model):
    exam = models.ForeignKey(
        Exam, null=True, on_delete=models.SET_NULL, related_name="categories"
    )
    name = models.TextField()
    # ...

class Question(models.Model):
    text = models.TextField()
    category = models.ForeignKey(
        Category,
        on_delete=models.SET_NULL,
        null=True,
        blank=True,
        related_name="questions",
    )
    exam = models.ForeignKey(
        Exam, null=True, on_delete=models.SET_NULL, related_name="questions"
    )
    # ...

On my frontend, I made a form where the user can create/edit an exam, and basically add/edit both categories and questions at the same time. Everything gets then sent at once in a POST/PUT request. The problem is, by the time everything is sent, categories haven’t been created in the db yet, so questions can’t reference them using their pk. In my frontend application, I temporarily use a UUID until I get an id from the db for the categories. The problem is I haven’t been able to find a good way to deal with the data on the server, as I can’t just use say a PrimaryKeyRelatedField on my question serializer to reference the category.

I’ll show the solution I’ve been using (which works fine but is ugly), and I’d very much appreciate it if someone could help me find a better way to achieve this, or even just give me some intuition as to what a better approach could look like.

What I did is I added a UUID field to Category: tmp_uuid. This holds the UUID temporarily generated in the frontend to reference the not-yet-created category.

The serializer for Question then gets added a field: category_uuid, which is write_only and only used to temporarily hold the relation to the category.

The idea is to, inside the update method of ExamSerializer, first create all the categories, then use the temporary UUID to get the questions referencing those categories, and build the permanent relation using the actual foreign keys.

This is the code:

def update(self, instance, validated_data):
        # get data about categories and questions
        questions_data = validated_data.pop("questions")
        categories_data = validated_data.pop("categories")

        instance = super(ExamSerializer, self).update(instance, validated_data)

        questions = instance.questions.all()
        categories = instance.categories.all()

        # update each category
        for category_data in categories_data:
            if category_data.get("id") is not None:
                category = Category.objects.get(pk=category_data["id"])
                save_id = category_data.pop("id")
            else:
                category = Category(exam=instance)
                category.save()
                save_id = category.pk

            serializer = CategorySerializer(
                category, data=category_data, context=self.context
            )
            serializer.is_valid(raise_exception=True)

            # update category
            serializer.update(instance=category, validated_data=category_data)

            # remove category from the list of those still to process
            categories = categories.exclude(pk=save_id)

          # remove any questions for which data wasn't sent (i.e. user deleted them)
        for category in categories:
            category.delete()

        # update each question
        for question_data in questions_data:
            if question_data.get("id") is not None: 
                question = Question.objects.get(pk=question_data["id"])
                save_id = question_data.pop("id")
            else:
                question = Question(exam=instance)
                question.save()
                save_id = question.pk

            serializer = QuestionSerializer(
                question, data=question_data, context=self.context
            )

            # pop question category as it's not handled by the serializer
            question_category = question_data.pop("category", None)

            # question belongs to a new category we just created
            if question_category is None:
                question_category = Category.objects.get(
                    tmp_uuid=question_data["category_uuid"]
                )

            serializer.is_valid(raise_exception=True)

            # update question
            updated_question = serializer.update(
                instance=question, validated_data=question_data
            )
            # update question category
            updated_question.category = question_category
            updated_question.save()

            # remove question from the list of those still to process
            questions = questions.exclude(pk=save_id)

        # remove any questions for which data wasn't sent (i.e. user deleted them)
        for question in questions:
            question.delete()

This is pretty contrived and not particularly elegant. The issue here is having to represent relations between objects that don’t yet exist.

Do you know any better way I could achieve this? Thank you!

2

Kind of bumping this thread up…

I have done some research and found out that a common approach in API’s is to disallow the creation of deeply nested data. Instead, you have to create the bits one by one (in my case, I would create the Questions separately) and then you link them together via pk. In other words, you can read nested data, but you can’t write it all at once. I imagine the reason is the difficulty in implementing the creation protocol like I faced in my case.

Stripe, for example, seems to do this. At this stage in the development of my app, I can’t really make such a big change–that would require rewriting a substantial amount of the code base. Do you have other suggestions?

3

I’m having trouble trying to visualize what you would be receiving from the front end. Do you have a sample JSON object (or two) that you can post showing what’s coming across the wire?

I’ll point out that making that type of change does not necessarily require a substantial rewrite - There’s nothing preventing the client from making multiple submissions of data sequentially, using an earlier response to populate part of a subsequent submission.

4

So, imagine I open the editor to update an exam that already exists. I’ll be presented with whatever categories and questions already exist. I might want to add new questions and categories, and send everything in a single PUT request. That request will update existing objects, create new ones, and delete those that were deleted in the frontend (that is, they exist on the db but no data was sent for them).

In the below request, imagine a category and a question already existed. In the session that results in the below request, I created a new category and a new question, and I put the question into the new category. This will show you both the creation of questions and categories, but you can imagine adding a new question to an already-existing category or creating a category and adding already-existing questions to it.

{
  "id": 2,
  "name": "test exam",
  "categories": [
    {
      "id": 5, // this category exists on the db so we can reference it by id
      "name": "oldCategory",
    },
    {
      "tmp_uuid": "99658163-886b-4877-ba55-f0f6135a6b82", // this category doesn't yet exist so it doesn't have an id, but it has a uuid because it's needed by questions (see below)
      "name": "newCategory",
    }
  ],
  "questions": [
    {
      "answers": [/* ... */ ],
      "text": "this question already exists",
      "id": 3,
      "category": 5 // belongs to a category that already exists on the db so we reference it by id
    },
    {
      "answers": [ /* ... */ ],
      "category_uuid": "99658163-886b-4877-ba55-f0f6135a6b82", // belongs to a category yet to be created, so we hold on to it using the uuid generated by the frontend
      "text": "this question doesn't exist on the db yet - we just created it in the frontend",
    }
  ]
}

The first thing the server will do is process the category. When it finds a category that doesn’t have an id but has a tmp_uuid, it’ll create it and store that uuid in a field in the category model. When it gets to the questions, if the question has a category field, it’ll look up the category by pk. In the case of the above request, the second question doesn’t have an id field, so first thing the server will do is it’ll create a new question. After that, it’ll notice there is no category field, but there is a category_uuid one. It’ll look up the (newly created) category by that uuid, and link it to the question via fk.

It’s very contrived, but it’s the only thing I could come up with. This way you can handle any combination of creating/deleting/updating questions and category at the same time.

5

Given your update request’s JSON shape is:

{
    ... exam properties
    questions: [ ... list of all questions ],
    categories: [ ... list of all categories ],
}

Could you change this to be:

{
    ... exam properties
    categories: [ 
        {
            ... category properties,
            questions: [ ... list of questions for this category]
        },
        ... other categories
    ],
}

This would avoid the temporary id for new categories. You’d already be aware of the questions for each category because it’s within the category dict.

2 Likes