A new trivial migration causes the previous migration to fail

Using Django Mystery Errors
1

I am somewhat new to django (~1 year) and was finally starting to feel pretty confident with myself until I ran into this issue which has left me at a loss.

My starting point had 15 migrations for my app. The 15th migration loaded data into my ‘plan’ table using the loaddata command.

At this point everything is A-OK. I can delete my database and migrate from scratch, app works great.

Now I did some work and noticed my tests would not run… giving the error
sqlite3.OperationalError: no such column: recommended_for_text
but my app was working fine, and isolated it to the smallest change that would cause the error:

I add a new (nullable) field to my plan model called ‘recommended_for_text’
I makemigrations, and migrate.

Having just done this, my app still works perfectly, however the unit tests wont run.

So I blow away my db and migrate and now it fails applying migration 0015 which worked before I created 0016.

It complains sqlite3.OperationalError: no such column: recommended_for_text

If i inspect the database as it exists now in its partially migrated state, and the column does in fact not exist (which is what I would expect)

migration 0015 is trying to insert records into the plan table, and the records it is trying to insert do not reference the ‘recommended_for_text’ at all, because it should not exist yet.

The dependencies all look right to me, but something is expecting that column to exist and I’m not sure if I’m doing something wrong (probably) or i have run into a bug (probably not!)

2

Can you post the contents of migrations 14, 15, and 16 here? (I’m wondering if there’s possibly a “dependencies” issue causing a problem.)

3

14 was unrelated but named

0014_sitecommandlog_note.py

0015_populate_pantheon_plans

# Generated by Django 4.1.10 on 2024-04-03 14:02
from django.core.management import call_command
from django.db import migrations


def load_pantheon_plans(apps, schema_editor):
    call_command('loaddata', 'pantheon_plans.json', app_label='sites')


class Migration(migrations.Migration):

    dependencies = [
        ("sites", "0014_sitecommandlog_note"),
    ]

    operations = [
        migrations.RunPython(load_pantheon_plans),
    ]

pantheon_plans.json has data like this (note no ‘recommended_for_text’ field)

{
  "model": "sites.pantheonplan",
  "pk": 1,
  "fields": {
    "portal_plan_name": "Sandbox",
    "pantheon_plan_sku": "plan-free-preferred-monthly-1",
    "traffic_limits": "No public traffic",
    "annual_plan_charge": "0.00",
    "is_active": true
  }
},

0016_pantheonplan_recommended_for_text

# Generated by Django 4.1.13 on 2024-04-25 17:34

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ("sites", "0015_populate_pantheon_plans"),
    ]

    operations = [
        migrations.AddField(
            model_name="pantheonplan",
            name="recommended_for_text",
            field=models.CharField(blank=True, max_length=256, null=True),
        ),
    ]
4

Ok, now that I have looked at this in my lab, it seems pretty obvious to me now what’s going on.

The issue is that the loaddata that you’re trying to perform in migration 15 is trying to use the model as it is currently defined in the models.py file - which has the field defined in it.

So the models.py has the field, but it hasn’t yet been added to the database, and that’s what’s causing the error.

5

Are you saying it’s actually a Django bug rather than my error?

6

After some more reading, I would describe it as an understandable limitation of the migration process. See #24778 (Data Migration from Fixture) – Django for some more information.

Briefly, my take is that you should not be using loaddata in your migration.

The comment in the example at Data Migrations kind of addresses this issue. Quoting from that example:

def combine_names(apps, schema_editor):
   # We can't import the Person model directly as it may be a newer
   # version than this migration expects. We use the historical version.
   Person = apps.get_model("yourappname", "Person")

My interpretation from this is that loaddata isn’t going to work because you don’t have an effective means of passing this app registry to it, for the loaddata to know what the schema structure is at the time of this migration.

So, in order to do this, you would need to process the json file yourself. (A brief look at how the deserialize method works doesn’t give me any confidence that it would be of much help, but I could easily be wrong here.)

1 Like
7

Interesting, as I said I’m relatively new to Django.

I had come to believe (and maybe it was ChatGPT that made me believe, I can’t remember my source) that migrations with loaddata where THE way to bootstrap your app with fundamental data.

Is there a different practice I should be using for this sort of thing?

Appreciate all of your insight. I have used loaddata a few times but this was the first time the model has changed anywhere I used it.

8

Yes. The docs for Data Migrations show a process of creating the objects yourself within the data migration. The key is that you need to use the models as they exist at the time this migration is being performed, not as they’re currently defined in the models.py files.

9

Ok I’ll take a look into that on Monday, thanks for all of your help, was really banging my head against a wall on this one!

Enjoy your weekend