Beginner overwhelmed with nested forms

Using Django Getting Started
1

I’ve finished the Django tutorial, and thought I could jump into a basic aspect of my project for now to get started. But I’m getting really overwhelmed and could use some pointers.

So this part of the app is basically a tool to add json data to a longtext field in the database for a given row. This json would be a two level nested structure, like this example:

{
  "people": [
    {
      "name": "Alice",
      "age": 30,
      "books": [
        {
          "title": "To Kill a Mockingbird",
          "author": "Harper Lee",
          "year": 1960
        },
        {
          "title": "1984",
          "author": "George Orwell",
          "year": 1949
        }
      ]
    },
    {
      "name": "Bob",
      "age": 25,
      "books": [
        {
          "title": "The Great Gatsby",
          "author": "F. Scott Fitzgerald",
          "year": 1925
        },
        {
          "title": "Brave New World",
          "author": "Aldous Huxley",
          "year": 1932
        }
      ]
    }
  ]
}

What I envisioned was a page that starts with a form for one person and one book for that person. And there would be buttons to add more people and a button under each person for adding more books. Then submitting it to a view, and having Python turn it into JSON to save to the database. And also being able to load already created JSON to populate such forms.

Possibly in the future, I would store people and books as models to be able to select from already existing options or add new ones. But for now, I’m okay with just Forms to input raw text.

The way I did this before hoping to make a more polished Django project was a simple Python script. It would just keep prompting for all the data in a couple nested while loops, which worked okay, but I want something more capable.

It has been a huge mess trying to come up with a solution. I don’t know whether to use Forms or Formsets. I absolutely hate Javascript. I’ve never worked with any JS frameworks, so I’m not sure if that would help. Can anyone point me in the right direction?

1 Like
2

This type of client-side manipulations of a page requires JavaScript. It can’t be done solely from the server. Frameworks like HTMX or Alpine.js can help, but you’re still going to find it worthwhile to understand what’s going on below the level of those frameworks.

And yes, multiple instances of a form generally implies that formsets are involved - that’s why formsets exist.

Finally, I’d suggest you don’t cut corners with your data schema. Design your data models to match your requirements. Learn how to work with relational data, don’t try to store everything as strings (or JSON).

1 Like
3

Thanks, yes I know I’ll need JavaScript. I’m just surprised this isn’t easier or a more common occurrence. Searching “dynamic nested forms” brings up far less (relatively understandable) information or libraries than I would have assumed.

My eyes just glaze over for any JavaScript over 20 lines. I don’t like the non linear nature when trying to conceptualize the flow.

Sure, that makes sense.

So for my example, would this mean to create two new models, Person and Book, with columns such as name, age, title. There can be repeat People in different rows of the main Model, and repeat Books in different rows of the People model. I’m not sure how to allow a Book to be tied to multiple People.

Other than a whole other Model just with connections between models, like: Moby Dick tied to Fred, Moby Dick tied to Susie.

4

It is a very common occurance. However, as this is a JavaScript issue and not (solely) a Django issue, the solution is going to be highly dependant upon your selection of JavaScript framework, and to a lesser degree, some of what you’re trying to do on the Django side.

And, in reality, it’s not that difficult, once you wrap your head around what needs to be done.

Side note: My general distaste for JavaScript is fairly well documented here - that’s why HTMX has been my tool of choice for a lot of what I do lately.

If a Book and be related to multiple People, and if a Person can be related to multiple Books, then that is the very definition of a Many-To-Many relationship. Django provides excellant support for them.

And yes, physically, that relationship exists in what I refer to as a Join table that consists of two ForeignKeys - one to each of the two models being related. This is a very standard structure in relational databases.

Side note: I would suggest you break this project down into smaller steps. Don’t try to solve everything all at once. Learn how to work with formsets (including dynamically adding forms for it) for the Person first. Once you have that working, and have gained an understanding of how formsets and the JavaScript all works, you’ll have a much easier time understanding how to add the books as a nested formset within each Person.

5

Since I’ve never worked with frameworks and know little about them, if I was to just get started, my first instinct would be to use one of the most common and popular ones like React. I’ve never heard of HTMX, is there any reason you think I should try that or another framework instead?

After thinking more about it, I realized I didn’t describe the relationships well enough. After brainstorming a database structure with ChatGPT, I drew this picture of it.

PXL_20231222_135728023~2
PXL_20231222_135728023~21920×1153 128 KB

Basically there would be a GroupModel for every main group, a PersonModel for every person, and a BookModel for every book. These would have data about the details of the item, like the person’s name or book title.

There would also be a GroupPersonModel, which would have foreign keys to both GroupModel and PersonModel, and would represent a person’s membership in a group. The reason for this intermediate model would be twofold: So that details can be added that specifically relate to this connection, for example, what color shirt the person wears to this specific group. And so that a book can connect to this Person-Group connection, representing a book used by this person in this group. I can’t just connect it to the person, because it needs to show what group the person is in when they use this book.

Similarly BookModel and GroupPersonModel would be connected with an intermediate model (GroupPersonBookModel) so that specific details about the copy of the book being used by this specific person in this group could be saved in this intermediate model, such as age of the book copy.

Maybe I just described a manually created version of the many-to-many Django relationship. Although I’m not sure if extra data like “color of shirt” can be added to the Join table you describe.

Any suggestions on how I would use formsets in this scenario? Ignoring books for now, and GroupModel doesn’t need to be changed, would I create one Formset based on PersonModel to create/change their name, and another Formset for GroupPersonModel to create/change their shirt color?

Good advice. I have a habit of being impatient and wanting to skip over the basics and learning.

6

My issue with React (or the other “major” frameworks such as Vue or Angular) is that my impression of them is that they’re designed to drive your application. In other words, they become the design around which your other components are created.
On the other hand, HTMX has worked for me very well as a “Django add-on”. I still feel like I’m working with Django, doing Django-ish things, and HTMX just helps by removing 90% of my need to write JavaScript.
Also, unlike those other frameworks, there’s no “build / integration” process to add to my workflow. It’s very much a situation of “add references to these files in my HTML and it works.”

So my conclusion has been that if you want a full SPA and want your development to be centered on JavaScript work and the JavaScript ecosystems, then you are better off going with Vue or React. On the other hand, if you’re looking for a more blended environment that can work like an SPA, without being heavily JavaScript oriented, then I would suggest taking a look at HTMX.

Yep, that’s exactly what you have described. And yes, you can add additional data to the join table. See the docs for Extra fields on many-to-many relationships

However, I would question whether Book is truly related to Book-GroupPerson. (Yes, you can do this, but it “smells bad” to me. However, I don’t understand your true requirements to know if there is a better recommendation to make.)

You typically use a formset for each model to be edited. When the model is related to another model, you would use an inline formset to manage that relationship. In other words, in your Group-Person situation, you could create formsets for Group, where each formset contains forms for all Group related to one Person. That formset could also contain fields to update the data in the Group-Person join table in addition to the fields for Group.

7

Ok, thank you for the insight. I’ll look into the various options, but I would prefer to have the focus be on Django/Python.

Oh, perfect. As long as I can have a many-to-many between another table and the first many-to-many.

Ok, I might as well describe the actual project instead of a toy example. The whole thing is basically for training an AI to detect internet comments where someone says that a health condition of theirs was treated or cured by a specific treatment. And then training it to extract the specific details, like treatment name and condition name.

The part of the project I’m describing above is an annotation tool for training the extraction AI. The training input will be the internet comment, and the training output will be something more or less like this:

{
  "treatments": [
    {
      "name": "Zoloft",  
      "hierarchy": ["medication", "antidepressants", "SSRIs"],
      "details": {
        "dosage": "50 mg daily",
        "duration": "6 months and ongoing", 
        "side_effects": "Some nausea in the first week"  
      },
      "helped_conditions": [ 
        {
          "condition": "depression",
          "confidence_score": 0.95,
          "helpfulness_score": 0.8
        }
      ]
    },
    {
     "name": "Cognitive behavioral therapy",
     "hierarchy": ["therapy", "cbt"], 
     "details": {
       "sessions": "Weekly 1 hour sessions"  
     },
     "helped_conditions": [
       {
         "condition": "anxiety",
         "confidence_score": 0.9,
         "helpfulness_score": 0.85
       }
     ]
    },
    {
      "name": "Fake it till you make it mindset",
      "hierarchy": ["lifestyle change", "mindset shift"],
      "helped_conditions": [
        {
          "condition": "low confidence",
          "confidence_score": 0.8,
          "helpfulness_score": 0.75  
        }
      ]
    }
  ]
}

So I need to create this json for a bunch of example internet comments. The tool would show one comment (corresponds to GroupModel). I could add the treatment name (PersonModel) and the specifics of the treatment in this specific scenario, like dosage (GroupPersonModel). Then each treatment corresponds to one or more conditions. I’d fill in the condition name (BookModel) and the specifics of the condition for this specific treatment in this scenario (GroupPersonBookModel). For example, the data that would go in this last many-to-many table would be the “helpfulness score”, which only applies to this specific relationship of condition-treatment-comment.

I would just add custom fields in the Form for this part, then deal with the submitted data myself?

8

Thank you - that helps a lot in my understanding and may help me better craft future responses.

Correct. (I think I’d generally prefer the term “additional field” instead of “custom field”, but that may just be me.)
When you’re processing the formset, you can save the formset for the model, then update the related objects from the information in those additional fields.

1 Like
9

So I ran into an issue with making a Formset for Treatments. From what I’m seeing online, it’s difficult to do inlineformset_factory with a many-to-many relationship. Here is my code, and the error I get:

models.py

class Treatment(models.Model):
    name = models.CharField(max_length=255)

    def __str__(self):
        return self.name

class TrainingText(models.Model):
    text = models.TextField()
    source = models.CharField(max_length=100)
    metadata = models.TextField(default=None, null=True)
    simple_label = models.IntegerField(default=None, null=True)

    treatments = models.ManyToManyField(Treatment, through="TrainingTextTreatment")

    def __str__(self):
        return self.text[:30] + "..." if len(self.text) > 30 else self.text

class TrainingTextTreatment(models.Model):
    training_text = models.ForeignKey(TrainingText, on_delete=models.CASCADE)
    treatment = models.ForeignKey(Treatment, on_delete=models.CASCADE)

    dosage = models.CharField(blank=True, null=True, max_length=255)

forms.py

class TreatmentForm(forms.ModelForm):
    class Meta:
        model=Treatment
        fields=["name"]

TreatmentFormSet = inlineformset_factory(
    TrainingText,
    Treatment,
    form=TreatmentForm,
    extra=1,
    can_delete=True,
)

ValueError: 'common.Treatment' has no ForeignKey to 'common.TrainingText'.

Do you have any suggestions for the best way to proceed?

Edit: I found this Stackoverflow answer that seems to apply. Not quite sure I understand it, or what makes an inlineformset different from a modelformset, but I’ll try to dig in.

Edit 2: I’m not sure the above answer applies, but for now I just used a basic modelformset_factory, and populated it with all the treatments that are tied to the trainingtext_id in the url. This is very satisfying:

Screenshot from 2023-12-22 18-44-10

10

(I want to acknowledge seeing your edit #2 - it looks like you have a solution that works for you - great!)

I had to go back and check some of my code.

In the best case that I could find, I see that what I had done was create the inline formset using the base table and the through table, and then create the additional fields to edit the related table.

Briefly:

class Author ...
    books = ManyToManyField('Book', through='BookAuthor')

class Book ...
    title = CharField(...)

class BookAuthor ...
    author = ForeignKey(Author, ...)
    book = ForeignKey(Book, ...)

author_formset = inlineformset_factory(Author, BookAuthor, form=BookAuthorForm ...

Then, the form initializes the book’s title field in the form initialization:

class BookAuthorForm ...
    title = form.CharField(...
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.fields['race'].initial = kwargs['instance'].book.title

(and then the save method on the form also needs to update and save the Book instance)

Then the view would have:

formset = author_formset(instance=Author.objects.get(id=...) )

(Actually, I think what I find most amazing is that the project I pulled this from was something I had done about 8 years ago…)

11

Not exactly a solution, just got the correct forms and data to display. Still have to try to tackle saving data and dynamically adding more forms.

I’m not sure I like having to custom add fields in the form. It seems like it might be easier for me to just do a modelformset and handle the logic of connections to the AuthorModel myself. I might have to think about your code a bit more though, as I don’t completely understand it.

Edit: Now that I think about it, the only field in Book will be the title, while the BookAuthor field will have many fields. So this might actually be best.

Also, ChatGPT said this, I was wondering what in the world “race” was:

self.fields[‘race’].initial = kwargs[‘instance’].book.title: This line seems incorrect as per the given context. It should probably be self.fields[‘title’].initial = kwargs[‘instance’].book.title if the goal is to initialize the title field with the title of the book related to the BookAuthor instance.

Yeah, amazing that you’re organized enough to find 8 year old relevant code.

12

Thanks, this is pretty much what I did, and it’s working great. Only difference is instead of the custom field for the name of the treatment in TrainingTextTreatmentForm, I just put it in `fields=[‘dosage’, ‘treatment’] so that you can select from all existing treatments in the form. And I made a second form for adding new treatments.

I think the forms for the second nested level (Condition many-to-many with the intermediate table TrainingTextTreatment) will be able to be made in a similar way.

And I don’t want to deal with dynamic forms right now, so I’ll just have each formset have one extra field to submit one new item at a time.

Here’s how it looks, being able to set treatments and dosage for the given text, and add new treatments to the database, so they are available to be added to this and other texts:

Screenshot from 2023-12-23 22-19-52
Screenshot from 2023-12-23 22-19-52626×679 50.3 KB