Help with many to many formsets

Using Django Forms & APIs
1

I am struggling with getting a usable many to many formset working – I have it largely functional, but from a UI perspective it is unusable, because it shows a select list with 16,000 objects in it (ie DirectoryLink3 object (32) is in the list, rather than Take away shops).

You can imagine that getting/formatting a select with 16,000 objects, multiple times, is a little slow, not even considering the time it takes to render in a browser.

As it is the forms save/delete formsets as expected. However, in this case I do not want a select list for each form, I just want the description field from the DirectoryLink3 model as a label/text (not input) and a delete checkbox for each form in the formset.

That is to say that the user won’t be able to edit the description, just delete the row. To add a row I have a separate autocomplete input categories, this uses javasccript to populate the required hidden form elements for a new row. The categories autocomplete seems to be working as expected, although I have yet to test it actually adding a record.

So, in breif: how to get the description field data without 16,000 siblings, show it as text on the form and control what ever other elements are in the object (ie do not show them).

models.py
   …
    user = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
   …
    directory_optin = models.BooleanField(_('Directory opt-in'), default=False, db_index=True)
    directory_desc = models.CharField(_('Business description'), max_length=512, default='', blank=True, null=False, db_index=True)
    directory_classes = models.ManyToManyField('DirectoryLink3', through='directorylink_profiles' )


class DirectoryLink3(models.Model):
    id = models.BigAutoField(primary_key=True)
    profiles = models.ManyToManyField('Profile', through='DirectoryLink_profiles')
    uplink = models.CharField(max_length=16, blank=True, null=True, db_index=True)
    catid = models.CharField(max_length=16, blank=True, null=True, db_index=True)
    description = models.CharField(max_length=128, blank=True, null=True, db_index=True)
    createdate = models.DateTimeField(auto_now_add=True)
    moddate = models.DateTimeField(auto_now=True)

    class Meta:
        db_table = 'directorylink_3'

class DirectoryLink_profiles(models.Model):
	id = models.BigAutoField(primary_key=True)
	directorylink = models.ForeignKey(DirectoryLink3, null=True, on_delete=models.CASCADE)
	profile = models.ForeignKey(Profile, verbose_name=_('profile'), null=True, on_delete=models.CASCADE)


views.py

class DirectoryUpdateView(UpdateView):
	template_name = 'directory/edit_directory.html'
	form_class = DirectoryForm
	model = Profile
	success_url = reverse_lazy('directory:directory-edit')

        #tried this, did not seem to work
	#formset_class = inlineformset_factory(Profile, DirectoryLink_profiles, fields=('directorylink', 'profile'))
	
	#required because there is no pk or slug
	def get_object(self, queryset=None):
		return Profile.get_or_create_for_user(self.request.user)

	def form_valid(self, form):
		print('form: ', dir(form))
		if form.data['directory_desc'] == '':
			messages.warning(self.request, 'The ‘directory description’ field is empty, \
				this may not be an error, but it is more useful to complete this field.')
			#form.add_error('directory_desc', 'Hello')
		self.object = form.save()
		messages.success(self.request, 'Changes to item have been saved')
		
		context = self.get_context_data(form=form)
		formset = context['directorychoice']
		if formset.is_valid():
			response = super().form_valid(form)
			formset.instance = self.object
			formset.save()
			return response
		else:
			return super().form_invalid(form)
		
	@method_decorator(login_required)
	def dispatch(self, *args, **kwargs):
		#print('self.formset: ', self.formset)
		log_views(self)
		return super(DirectoryUpdateView, self).dispatch(*args, **kwargs)
		
	def get_context_data(self, **kwargs):
	
		context = super(DirectoryUpdateView, self).get_context_data(**kwargs)
		
		if self.request.POST:
			context['directorychoice'] = DirectoryChoice(self.request.POST, instance=self.object)
			context['directorychoice'].full_clean()
		else:
			context['directorychoice'] = DirectoryChoice(instance=self.object)
		return context


forms.py

class DirectoryForm(forms.ModelForm):

    directory_desc = forms.CharField(label=_('Directory description'), widget=forms.Textarea(attrs={'rows':5, 'blank': True}), required=False, help_text=_('Give a short description of your business, up to 512 characters') )
    directory_list = forms.CharField(label=_('Categories'), help_text=_('Type a few characters and select from the pop-up menu'), required=False)

    class Meta:
        model = Profile
        
        fields = (
            'directory_optin',
            'directory_desc',
            'directory_list',
        )

DirectoryChoice = forms.models.inlineformset_factory(
							Profile, 
							DirectoryLink_profiles, 
							fields = ['id', 'directorylink', 'profile'], 
							exclude = [], 
							can_delete = True,
							max_num=3,
							extra=1,
							)


edit_directory.html

…
{{ directorychoice.management_form }}
…
{% bootstrap_formset directorychoice layout='horizontal' %}
…

Sorry, that posted before I’d finished.

2

inlineformset_factory takes a named parameter form, which would be a ModelForm class (not instance!) for use by the factory function. You can create your form class to be whatever you want it to be - custom widgets, fields, whatever.

See Specifying widgets to use in the inline form and the ModelForm factory function docs.

3

Ken, I’m taking it that you mean this line here (and the one just below it):

			context['directorychoice'] = DirectoryChoice(self.request.POST, instance=self.object)

I took this from someone else’s example, which seems to work.

I believe I tried several ways to use a form but could not get it to work.

However, I have come to the realisation that, whatever widget I apply to a field, I am still going to end up with 16,000 of them per form (ie I’ll get 16,000 in a select, 16,000 checkboxes or 16,000 text input fields), which is absolutely no use to anyone.

With that in mind I have a plan for a solution, which I’ll post if/when I get it working.

I also realised that it isn’t the retrieval of 16,000 records that is bringing the proces to its knees (the query and the return of the data is nearly instantaneous), it’s django’s processing and formatting, plus the browser churning through around 7MB of HTML. So, important to reduce that load.

4

Actually, I was referring to this line:

You can define a form class with whatever structure you desire and pass it to the factory using the form parameter.

DirectoryChoice = forms.models.inlineformset_factory(
    Profile, DirectoryLink_profiles, form=CustomModelForm, ...
)

where CustomModelForm is a Form class you create to build the form you want used in the formset.

5

Oh, I got that part. Like I wrote, I couldn’t get that to work as expected. But I’ll give it another go when I get to that part again.

The bit I was referring to was where you wrote:

(not instance!)

But I see what you mean now.

I take it you agree that the widget type does not matter, I’m still going to get 16,000 of them?

6

No, my understanding from what you wrote here:

So you wouldn’t necessarily use a widget - just a text field with the desired description (unless I’m getting my fields confused from your example - in which case I apologize for my confusion.).

You don’t need to use any type of selection box for a field, and if it’s a non-editable field, you wouldn’t need to use a widget at all - just use the value as a label. (If you wanted it to look like a widget, you could use a normal text-entry box as the widget with the value as the default value and make it disabled / non-editable.)

That’s why a custom form may be useful here - you will have full control over what’s rendered and how.

(You can also create a view to test your new form, to ensure that it’s going to render correctly, before applying it to your formset - I’ve found it helpful in the past to do that.)

Note, if you do want it to be an editable field, there are also ajax-based “auto-complete” fields that greatly reduce the size of the initial form at the expense of the ajax calls to retrieve partial results.

7

I think that it doessn’t matter if it is an imput type or a label, there are still going to be 16k instances – I looked at the queries, there is no WHERE clause, so it is getting everything in the table.

Ajax is the obvious solution to populating new rows, this is what I mention in my original post – this complex piece of UI is actually amongst the easiest parts of the problem to solve!

Thanks for the feedback.

8

I’m curious, you’ve lost me here. 16K instances of what? Are you saying you’re generating 16K rows in your formset? (I was under the impression that you’re talking about 16K choices within a single select field, in which case changing the select field to a text box in the form is not going to generate 16K elements, because you’re not going to query on the entire table - you’re retrieving a single value in your form for that box.)

9

No, you’re right. I got the form input in the inlineformset_factory working, then when the directorylink field is set to the TextInput widget the entire query to get the data for the formset changes.

My form looks like this:

class DirectoryChoices(forms.ModelForm):
	
	class Meta:
		model = DirectoryLink_profiles
		
		fields = (
			'uselesstext',
			'profile',
			'directorylink',
			)
			
		
		widgets = {
            'directorylink': forms.TextInput(attrs={'placeholder': 'Do not edit!'}),
        }

Following which, my only problem is getting the description field on the form, where the directorylink field is the foreign key to the DirectoryLink3 model.

In SQL terms:

select DirectoryLink3.description from DirectoryLink_profiles
join DirectoryLink3 on DirectoryLink_profiles.directorylink = DirectoryLink3.id
where id = …
10

Actually, I just got done posting these notes on this on another thread -
Briefly:

  • Remove directorylink from the list of fields - you don’t want Django generating that form field
  • Define a field for the description in the form.
  • Override the __init__ method to set the field if the form is bound to an object
  • Override the save method to save that field in the related object (if you’re updating it)

Keep in mind that a ModelForm is a form, with the additional functionality of Django creating form fields for you in a more “declarative” mode. So anything you could do in a form, you can also do in a ModelForm. A ModelForm doesn’t remove any functionality from a Form.

11

Ok, I think I am pretty close. The description field appears on the formset, but is empty. I print the content I expect to see in the description in the terminal, and that looks exactly as I expect – so a bit confused as to how I get these values on the browser page.

class DirectoryChoices(forms.ModelForm):
	model = DirectoryLink_profiles
	
	def __init__(self, *args, **kwargs):
		
		super().__init__(*args, **kwargs)
	
		if 'fields' in dir(self) :
			self.base_fields['description'] = forms.CharField()
			self.fields['description'] = forms.CharField()
			self.fields['description'].initial = 'this is hard work'
			print('fields: ', (self.fields['description'].initial))
		
			self.fields['description'].initial = kwargs['instance'].directorylink.description
			########################################
			print('self.fields[description]: ', (self.fields['description'].initial))
			########################################

		print('kwargs yourself dir: ', dir(kwargs))
		print('kwargs yourself dict: ', dict(kwargs))
		#initial = kwargs.get('initial', {})
		#self.fields['description'] = forms.CharField()
		self.fields['description'] = 'hell, initial description'
		#description = 'hell, initial description'
		
		super(DirectoryChoices, self).__init__(*args, **kwargs)
	
	class Meta:
		model = DirectoryLink_profiles
		
		fields = (
			'uselesstext',
			'profile',
			)
			
		widgets = {
            'directorylink': forms.TextInput(attrs={'placeholder': 'Do not edit!'}),
        }

If I put description in the fields list I get an error that the field is unknown.

12

You are really close - you don’t want to change the “initial” attribute on the field, but you do want to change the initial dict on the form, with the key of the field name.
e.g. If your form field is description, then after calling super().__init__(...), you can the initial value using:
self.initial['description'] = "some appropriate value"

13

Actually seems you can do it both ways:

self.fields['description'].initial = kwargs['instance'].directorylink.description

and

self.initial['description'] = kwargs['instance'].directorylink.description

That was not the cause of empty description fields, rather, this was, at the bottom of ___init__:

super(DirectoryChoices, self).__init__(*args, **kwargs)

Take that out and the fields are populated.

Thanks for the feedback.

1 Like