Is there way to override changed data in django forms?

mrchrisadams · October 2, 2023, 5:49pm

hi folks.

I think I need some help thinking through how to implement some fairly gnarly form logic, and before I spend more time banging my head against a wall, I figured asking here might help.

Background

I’m working on an open source project, where we’re building a multi step form wizard that it takes some data from a series of other models, and allows users to run through a series of steps to confirm that data is either correct, or needs to be updated, before writing the updated version back to the original model.

We’ve been using the initial_data method in django forms to prepopulate forms and formsets with data to allow a user to review it before either making the changes or accepting them as they are.

To make this concrete, you can see some code at the link below, in the class ProviderRequestWizardView, and in particular, the code to generate our initial_data using get_initial_dict() on that object.

github.com

thegreenwebfoundation/admin-portal/blob/oz-edit-hosting-provider/apps/accounts/views.py#L597


      
                  cls.Steps.NETWORK_FOOTPRINT.value: {
                      "ips": ip_qs,
                      "asns": asn_qs,
                      "extra": pr_instance,
                  },
                  cls.Steps.CONSENT.value: pr_instance,
              }
              return instance_dict
          
          @classmethod
          def get_initial_dict(cls, provider_id):
              """
              Based on provider_id, return data about existing Hostingprovider
              in a format expected by the consecutive forms of WizardView
              """
          
              def _evidence_initial_data(evidence: HostingproviderCertificate):
                  return {
                      "title": evidence.title,
                      "description": evidence.description,
                      "link": evidence.url,

And later on using the done

github.com

thegreenwebfoundation/admin-portal/blob/oz-edit-hosting-provider/apps/accounts/views.py#L340


      
                      )
                      return HttpResponseRedirect(
                          reverse("provider_request_detail", args=[pr.pk])
                      )
                  # only admins and creators can access for editing existing requests
                  if not request.user.is_admin and request.user.id != pr.created_by.id:
                      raise Http404("Page not found")
          
              return super().dispatch(request, *args, **kwargs)
          
          def done(self, form_list, form_dict, **kwargs):
              """
              This method is called when all the forms are validated and submitted.
          
              Here we create objects for ProviderRequest and related models,
              based on the validated data from all Forms
              (passed as `form_dict`, where keys are the names of the steps).
          
              Because this method is called via POST request,
              it must return a redirect to a DetailView
              of a created ProviderRequest instance.

One thing we’re finding though is that when we use django forms, only the data that has changed is being saved as we run through the wizard, when we have initial data.

This wizard works fine for when we use it to allow a user to submit entirely new information, (i.e. to run through a form wizard with no initial data being added), but when we’re workign with existing data, this is bad because at the end of the wizard, when we would save the information back to the original models, only the data that has been changed is persisted back to the models.

In our case, we want all the data to be saved though, no matter what - is there a simple way to override this behaviour so we do this?

mrchrisadams · October 2, 2023, 5:54pm

Adding a bit more here. I initially thought this was a problem related to the has_changed behaviour with django forms.

Use the has_changed() method on your Form when you need to check if the form data has been changed from the initial data.

Source: Django

This made me think I could override that.

However, I’m now thinking it’s more related to our use of formsets, because of these two paras in the docs - first this:

Validation with a formset is almost identical to a regular Form. There is an is_valid method on the formset to provide a convenient way to validate all forms in the formset:

… and later this:

The formset is smart enough to ignore extra forms that were not changed

Source: Django by djangoproject

I think in our case, we don’t want to ignore forms that have initial data that were not changed - we want to save them too, because if the data has not been changed, then it means we do want it still.

Is there an straight forward day to override is_valid to not ignore these forms, so that when saving a formset, we persist all the data in the form, without filtering out any unchanged forms?

Thanks

mrchrisadams · October 2, 2023, 6:15pm

Ok, this looks like the method that might be filtering out the unchanged forms:

github.com

django/django/blob/main/django/forms/models.py#L777-L793


      
          def save(self, commit=True):
              """
              Save model instances for every form, adding and changing instances
              as necessary, and return the list of instances.
              """
              if not commit:
                  self.saved_forms = []
          
                  def save_m2m():
                      for form in self.saved_forms:
                          form.save_m2m()
          
                  self.save_m2m = save_m2m
              if self.edit_only:
                  return self.save_existing_objects(commit)
              else:
                  return self.save_existing_objects(commit) + self.save_new_objects(commit)

If looks like save_existing_objects and save_new_objects both drop a form if form.has_changed() is false:

github.com

django/django/blob/main/django/forms/models.py#L918-L956


      
          def save_existing_objects(self, commit=True):
              self.changed_objects = []
              self.deleted_objects = []
              if not self.initial_forms:
                  return []
          
              saved_instances = []
              forms_to_delete = self.deleted_forms
              for form in self.initial_forms:
                  obj = form.instance
                  # If the pk is None, it means either:
                  # 1. The object is an unexpected empty model, created by invalid
                  #    POST data such as an object outside the formset's queryset.
                  # 2. The object was already deleted from the database.
                  if obj.pk is None:
                      continue
                  if form in forms_to_delete:
                      self.deleted_objects.append(obj)
                      self.delete_existing(obj, commit=commit)
                  elif form.has_changed():

This file has been truncated. show original

This makes me think we’ll need override save_existing_objects, to avoid dropping unchanged forms for this to work.

I’m currently not sure what the full implications of this might be - we’d presumably save more things to the database rather than just the bits that have changed, so that might trigger more calls to save().

In our case, I think that’s manageable.

KenWhitesell · October 2, 2023, 6:36pm

<conjecture>
I believe one of the biggest implications would involve any “extra” forms that may have been generated and rendered in the formset. If you generate an extra form and it’s submitted without data, you probably don’t want that attempted to be validated or saved.

If you are 100% certain that you can guarantee that no “extra” forms will be generated in the formset and that the front-end will never provide the facility to add forms dynamically, you’re probably ok with this.
</conjecture>

As a side note, my initial reaction is that I’m not sure I understand why not saving those unchanged forms is a problem. If you have the data when the form is being rendered, then you still should have that data available when the form is posted.

carltongibson · October 4, 2023, 7:49am

Hey @mrchrisadams

So just checking, you’ve looked at the form tools package right?

https://django-formtools.readthedocs.io

This provides a form wizard object that has likely covered most of the issues your facing here, at least in a manner you might draw from, even if it’s not perfect for your case.

The links you posted seem on the right path (but it’s difficult to say exactly without looking at depth in your code). You should be able to store (in the session?) a record of which forms were from a previous step (and should not be treated as unchanged).

One thought that comes to mind (but literally just so … ) is to merge previously submitted data into the data you pass to the formset, so from it’s POV it all got submitted in one go. I have no idea how easy or not that might be without trying, but it might be only a ChainMap (although getlist()…)

mrchrisadams · October 4, 2023, 7:58pm

hi @carltongibson - yes we’re using the form-tools package extensively - it’s been fairly nice to work with, and we expect to work with it more in the future.

You can see the specific implementation below, where we’re extending the SessionWizardView

github.com

thegreenwebfoundation/admin-portal/blob/master/apps/accounts/views.py#L233


      
              def get_queryset(self) -> "QuerySet[ProviderRequest]":
                  """
                  Admins can retrieve any ProviderRequest object,
                  regular users can only retrieve objects that they created.
                  """
                  if self.request.user.is_admin:
                      return ProviderRequest.objects.all()
                  return ProviderRequest.objects.filter(created_by=self.request.user)
          
          
          class ProviderRequestWizardView(LoginRequiredMixin, WaffleFlagMixin, SessionWizardView):
              """
              Multi-step registration for providers.
              - uses `django-formtools` SessionWizardView to display
                the multi-step form over multiple screens,
              - requires the flag `provider_request` enabled to access the view,
          
              """
          
              waffle_flag = "provider_request"
              file_storage = DefaultStorage()

I hadn’t thought about making a copy of the object like that to compare against before - if i have the ‘pristine’ versions of the data as an object before someone starts working through a multi-stage form wizard, then I wouldn’t need to reach into the formset validation logic like I described.

I think I’d just need to look at the changed forms in each formset of the wizard, and then swap them into the ‘pristine’ version of the object, replacing the now out-of-date versions.

That seems less hairy then messing with core django formset logic, and lets me keep the merge logic in one place.

Thanks for the pointers

Topic		Replies	Views
what data can I get when overriding a change_form Using Django	1	833	September 19, 2020
Model Form - Provide initial Values Using Django	0	801	April 5, 2020
UserCreationForm : Can't override class name ? Using Django	12	6413	July 2, 2020
Django forms - prepopulate a Boolean field.... Forms & APIs	4	959	June 19, 2023
How to get form field value before and after submit Using the ORM	5	72	September 1, 2024

Is there way to override changed data in django forms?

Background

Related topics