Why can't I filter on this related field anymore after upgrading Django versions?

I’m upgrading a project from Django 1.8 to Django 2.1.3 (yes, I know I should update more, but that’s the client requirement at this point). The exact same models.py file and exact same Django ORM query in the views.py file are in both (Python3 updates notwithstanding), the Django 1.8 version works fine but with Django 2.1.3 I’m seeing the following error:

raise FieldError('Related Field got invalid lookup: {}'.format(lookups[0]))

django.core.exceptions.FieldError: Related Field got invalid lookup: carcategory

Anybody know what gives here? The DB is the same, nothing’s been changed there, and like I said, the models.py file is exactly the same (aside from updating some ForeignKey fields with on_delete=models.CASCADE for the Django 2.1.3 version)

Here’s the main query:

cars = CarModel.objects.only(

'car__car_id',

'car__cardate',

'car__title',

'car__location',

'car__quote',

'car__key',

'car__cartype_id',

'maker__lastname',

'maker__firstname',

'maker__middlename',

'maker__nickname',

).select_related(

'car',

'maker',

).extra(

select={

'is_position_paper': 'cartype_id = 7',

'is_null_date': 'cardate IS NULL',

'shorttitle': extra,

},

).filter(**kwargs).distinct(sort_order_for_distinct, 'car__car_id').order_by(sort_order, 'car__car_id')

Prior to update Django version, on Django 1.8 I was able to add the following filter to the above query:

kwargs['car__carcategory__category__category_id'] = categoryId

This is the CarModel table:

class CarModel(models.Model):
    car_candidate_id = models.IntegerField(primary_key=True)
    car = models.ForeignKey(Car, on_delete=models.PROTECT)
    candidate = models.ForeignKey(Manufacturer, on_delete=models.PROTECT)
    created = models.DateTimeField(blank=True, null=True)
    modified = models.DateTimeField(blank=True, null=True)

    class Meta:
        managed = False
        db_table = 'car_model'

This is the Car table:

class Car(models.Model):
    car_id = models.IntegerField(primary_key=True)
    cartype = models.ForeignKey('Cartype', on_delete=models.PROTECT)
    title = models.CharField(max_length=200)
    class Meta:
        managed = False
        db_table = 'car'

This is what the Carcategory table looks like:

class Carcategory(models.Model):
    car_category_id = models.IntegerField(primary_key=True)
    car = models.ForeignKey(Car, on_delete=models.CASCADE)
    category = models.ForeignKey(Category, on_delete=models.CASCADE)
    class Meta:
        db_table = u'car_category'

This is the Category table:

class Category(models.Model):
    category_id = models.SmallIntegerField(primary_key=True)
    name = models.CharField(max_length=255)
    description = models.CharField(max_length=2048, blank=True)
    release_id = models.SmallIntegerField()
    key = models.NullBooleanField(null=True)
    rank = models.SmallIntegerField(null=True)
    class Meta:
        db_table = u'category'
    def __unicode__(self):
        return unicode(self.category_id)

Is that a copy/paste error or a search & replace error?

Sorry, that is a typo, I will update (I wish it was that simple…).

I’ve kept searching on this issue, I can’t find anything. I don’t know if something was deprecated or what, but the EXACT same code has no issues in Django 1.8, but is now totally borked. I just don’t get it. It’s refusing to see this carcategory table…

What processes did you take to update your django version?

I don’t see anything wrong in what you’ve posted, and the simple test that I put together with simplified models works fine.

If I had to debug this, I’d start by working this through the shell. Start from the smallest possible query that retrieves the data and then add clauses one-by-one until you find out where it breaks.

In the shell, it fails in the same place: with the filter lookup that used to work with Django 1.8

Is there any reason you can think of that you all of a sudden couldn’t do a lookup like that? Someone pointed me towards these links, which maybe might shed some light on this?

https://docs.djangoproject.com/en/3.1/releases/1.9/#implicit-queryset-in-lookup-removed

https://docs.djangoproject.com/en/3.1/releases/1.10/#select-related-prohibits-non-relational-fields-for-nested-relations

And these:

But I don’t know what to make of any of the above here.

Would the order of how I’m importing these models into my views.py file make a difference ? Ran a diff check on the relevant models.py files for each version, and they’re identical, save that in models.py for Django 2.1.3 the foreignkey fields all have on_delete=models.CASCADE in them now.

on_delete=models.CASCADE means that if the model connected by a foreign key or 1-1 is deleted, then any connected models will also be deleted. When you’re doing select_related in your query, you’re trying to query the car model connected to carcategory. Does the instance of car you’re trying to reference exist in the database?

In theory, it is possible that the order makes a difference if you’ve got different models by the same name in different files, but that’s a real edge case.

When I was talking about building it incrementally, if this is your target query:

cars = CarModel.objects.only('car__car_id', 'car__cardate', 'car__title',
 'car__location', 'car__quote', 'car__key', 'car__cartype_id',
 'maker__lastname', 'maker__firstname', 'maker__middlename', 
 'maker__nickname', 
).select_related('car', 'maker',
).extra(
  select={
    'is_position_paper': 'cartype_id = 7',
    'is_null_date': 'cardate IS NULL',
    'shorttitle': extra,
},
).filter('car__carcategory__category__category_id'
).distinct(sort_order_for_distinct, 'car__car_id'
).order_by(sort_order, 'car__car_id')

I’d actually first try:

cars = CarModel.objects.all(
).filter('car__carcategory__category__category_id'=categoryId
)

Just to make sure that everything is working fundamentally. If this works, then you can start adding the different components of the query until you find out what’s causing it to break and focus your search in that area.

It might help at this point to see the view, too.

1 Like

Hi Ken, thanks for all your help with this. I attempted what you said in the shell, just trying that simple query:

cars = CarModel.objects.all(
).filter(car__carcategory__category__category_id=categoryId
)

and no dice, I still get:

raise FieldError('Related Field got invalid lookup: {}'.format(lookups[0]))

django.core.exceptions.FieldError: Related Field got invalid lookup: carcategory

Yet, if I try:

foo = Carcategory.objects.all().filter(car_id=1101217)

There’s no issue, the object is there:

<QuerySet [<Carcategory: Carcategory object (3084631)>]>

So if I start from Carcategory, the relationship exists…but I can’t go backwards within the context of Carmodel for some reason?

Just for reference, this kind of query returns the expected results on the DB as it exists now in SQL:

SELECT * FROM car_model JOIN car USING (car_id) JOIN car_category USING (car_id) WHERE car.car_id=car_category.car_id

Ok, that does narrow it down a great deal. My next attempt would then be to try the intermediate step - Car.objects.all(carcategory__car_category_id=<some known value>) to see whether or not it’s seeing the reverse-relationship at all.

I’m wondering if this is a bug that was resolved in a later release. I created a model structure similar to yours and have no problems with that type of query in my test environment.

I did some digging through the old Django tickets, and only found one that appears to be close (sorry, lost track of the number). In that particular case, it was a user error in that they were using a module that changed an attribute on the model. That attribute name wasn’t used in older versions of Django but is being used now, so that module was overwriting what Django was using, causing Django to fail. (I believe the attribute is “field_name”)

I would also make quadrupally sure that I don’t have any other models anywhere in the application (or in any 3rd-party apps installed) with the same name as any of these.

I’m still looking for other possibilities…

Ken

1 Like

Thank you again Ken, ok it looks like something was shook loose here!

I tried this many times before, but for some reason Django picked up on this in the Carcategory model: I added the following related_name params to the speech and category fields (I gave a related_name that is long and obnoxious just to see if there are any maybe namespace issues or something?), so it looks like this:

class Carcategory(models.Model):
    car_category_id = models.IntegerField(primary_key=True)
    car = models.ForeignKey(Car, on_delete=models.CASCADE, related_name='carCategoryModelInGovernment')
    category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='carCategoryModelInGovernment')
    class Meta:
        db_table = u'car_category'

Now, if I try a slight variation on your test:

cars = Car.objects.prefetch_related('carCategoryModelInGovernment').filter(carCategoryModelInGovernment__category__category_id__in=[90]).count()

I get:

99661

Whereas if I just do:

cars = Car.objects.prefetch_related('carCategoryModelInGovernment').all().count()

I get:

1343170

So, Carcategory is seemingly accessible now, at least to Car model…So I thought this would work now, getting back to the original issue:

cars = CarModel.objects.prefetch_related('car', 'car__carCategoryModelInGovernment').all()

But no luck there, getting an invalid parameter to prefetch_related() error.

If I try this:

cars = CarModel.objects.prefetch_related('car').filter(car__carCategoryModelInGovernment__category__category_id__in=[90]).all()

I just get a django.core.exceptions.FieldError: Related Field got invalid lookup: carCategoryModelInGovernment

I tried chaining a prefetch_related('car__carCategoryModelInGovernment') to the select_related(), but still no dice.

So I can get the Car model to “see” the reverse relationship directly from itself, but not a relationship removed if I move back to CarModel --> Car --> Carcategory.